Shanker Blog

The Test-Based Evidence On The "Florida Formula"

** Reprinted here in the Washington Post

Former Florida Governor Jeb Bush has become one of the more influential education advocates in the country. He travels the nation armed with a set of core policy prescriptions, sometimes called the “Florida formula," as well as "proof" that they work. The evidence that he and his supporters present consists largely of changes in average statewide test scores – NAEP and the state exam (FCAT) – since the reforms started going into place. The basic idea is that increases in testing results are the direct result of these policies.

Governor Bush is no doubt sincere in his effort to improve U.S. education, and, as we'll see, a few of the policies comprising the “Florida formula” have some test-based track record. However, his primary empirical argument on their behalf – the coincidence of these policies’ implementation with changes in scores and proficiency rates – though common among both “sides” of the education debate, is simply not valid. We’ve discussed why this is the case many times (see here, here and here), as have countless others, in the Florida context as well as more generally.*

There is no need to repeat those points, except to say that they embody the most basic principles of data interpretation and causal inference. It would be wonderful if the evaluation of education policies – or of school systems’ performance more generally - was as easy as looking at raw, cross-sectional testing data. But it is not.

Luckily, one need not rely on these crude methods. We can instead take a look at some of the rigorous research that has specifically evaluated the core reforms comprising the “Florida formula." As usual, it is a far more nuanced picture than supporters (and critics) would have you believe.

Holiday Wishes From The Shanker Institute

We at the Shanker Institute are taking a break for the holidays.

The Year In Research On Market-Based Education Reform: 2012 Edition

** Reprinted here in the Washington Post

2012 was another busy year for market-based education reform. The rapid proliferation of charter schools continued, while states and districts went about the hard work of designing and implementing new teacher evaluations that incorporate student testing data, and, in many cases, performance pay programs to go along with them.

As in previous years (see our 2010 and 2011 reviews), much of the research on these three “core areas” – merit pay, charter schools, and the use of value-added and other growth models in teacher evaluations – appeared rather responsive to the direction of policy making, but could not always keep up with its breakneck pace.*

Some lag time is inevitable, not only because good research takes time, but also because there's a degree to which you have to try things before you can see how they work. Nevertheless, what we don't know about these policies far exceeds what we know, and, given the sheer scope and rapid pace of reforms over the past few years, one cannot help but get the occasional “flying blind" feeling. Moreover, as is often the case, the only unsupportable position is certainty.

The Sensitive Task Of Sorting Value-Added Scores

The New Teacher Project’s (TNTP) recent report on teacher retention, called “The Irreplaceables," garnered quite a bit of media attention. In a discussion of this report, I argued, among other things, that the label “irreplaceable” is a highly exaggerated way of describing their definitions, which, by the way, varied between the five districts included in the analysis. In general, TNTP's definitions are better-described as “probably above average in at least one subject" (and this distinction matters for how one interprets the results).

I’d like to elaborate a bit on this issue – that is, how to categorize teachers’ growth model estimates, which one might do, for example, when incorporating them into a final evaluation score. This choice, which receives virtually no discussion in TNTP’s report, is always a judgment call to some degree, but it’s an important one for accountability policies. Many states and districts are drawing those very lines between teachers (and schools), and attaching consequences and rewards to the outcomes.

Let's take a very quick look, using the publicly-released 2010 “teacher data reports” from New York City (there are details about the data in the first footnote*). Keep in mind that these are just value-added estimates, and are thus, at best, incomplete measures of the performance of teachers (however, importantly, the discussion below is not specific to growth models; it can apply to many different types of performance measures).

Are Teachers Changing Their Minds About Education Reform?

** Reprinted here in the Washington Post

In a recent Washington Post article called “Teachers leaning in favor of reforms," veteran reporter Jay Mathews puts forth an argument that one hears rather frequently – that teachers are “changing their minds," in a favorable direction, about the current wave of education reform. Among other things, Mr. Mathews cites two teacher surveys. One of them, which we discussed here, is a single-year survey that doesn't actually look at trends, and therefore cannot tell us much about shifts in teachers’ attitudes over time (it was also a voluntary online survey).

His second source, on the other hand, is in fact a useful means of (cautiously) assessing such trends (though the article doesn't actually look at them). That is the Education Sector survey of a nationally-representative sample of U.S. teachers, which they conducted in 2003, 2007 and, most recently, in 2011.

This is a valuable resource. Like other teacher surveys, it shows that educators’ attitudes toward education policy are diverse. Opinions vary by teacher characteristics, context and, of course, by the policy being queried. Moreover, views among teachers can (and do) change over time, though, when looking at cross-sectional surveys, one must always keep in mind that observed changes (or lack thereof) might be due in part to shifts in the characteristics of the teacher workforce. There's an important distinction between changing minds and changing workers (which Jay Mathews, to his great credit, discusses in this article).*

That said, when it comes to the many of the more controversial reforms happening in the U.S., those about which teachers might be "changing their minds," the results of this particular survey suggest, if anything, that teachers’ attitudes are actually quite stable.

Egypt's Islamist Government Reaches For Control Of The Unions

Our guest author today is Heba F. El-Shazli. Currently a visiting professor at Georgetown University’s Center for Democracy and Civil Society, Prof. El-Shazli  has 25 years of international experience in political and economic development, including democracy promotion programs and  support  for independent trade unions. She has worked with trade unions, political parties, and leaders throughout the Middle East and North Africa (MENA). The views expressed here are her own.

It is becoming uncomfortably clear that the strategy of the Muslim Brotherhood (MB) is to take control of many of Egypt’s major civil society organizations. While everyone’s attention is focused on the assault on the constitution and the judiciary, President Mohammed Morsi has strategically revised a trade union law that will affect millions of Egyptian workers.

If successful, this move will affect the lives, jobs, and political freedom of millions of Egyptian workers, as well as renew the Egyptian Trade Union Federation’s (ETUF) longstanding role as an enforcer of government labor policy.

Cheating, Honestly

Whatever one thinks of the heavy reliance on standardized tests in U.S. public education, one of the things on which there is wide agreement is that cheating must be prevented, and investigated when there’s evidence it might have occurred.

For anyone familiar with test-based accountability, recent cheating scandals in Atlanta, Washington, D.C., Philadelphia and elsewhere are unlikely to have been surprising. There has always been cheating, and it can take many forms, ranging from explicit answer-changing to subtle coaching on test day. One cannot say with any certainty how widespread cheating is, but there is every reason to believe that high-stakes testing increases the likelihood that it will happen. The first step toward addressing that problem is to recognize it.

A district, state or nation that is unable or unwilling to acknowledge the possibility of cheating, do everything possible to prevent it, and face up to it when evidence suggests it has occurred, is ill-equipped to rely on test-based accountability policies. 

Selective Schools In New Orleans

Charter schools in New Orleans, LA (NOLA) receive a great deal of attention, in no small part because they serve a larger proportion of public school students than do charters in any other major U.S. city. Less discussed, however, is the prevalence of NOLA’s “selective schools” (elsewhere, they are sometimes called “exam schools”). These schools maintain criteria for admission and/or retention, based on academic and other qualifications (often grades and/or standardized test scores).

At least six of NOLA’s almost 90 public schools are selective – one high school, four (P)K-8 schools and one serving grades K-12. When you add up their total enrollment, around one in eight NOLA students attends one of these schools.*

Although I couldn’t find recent summary data on the prevalence of selective schools in urban districts around the U.S., this is almost certainly an extremely high proportion (for instance, selective schools in New York City and Chicago, which are mostly secondary schools, serve only a tiny fraction of students in those cities).

Bahraini Repression Against Teachers And Health Care Workers

The Kingdom of Bahrain, a small island nation in the Persian Gulf, is a perfect political stew, situated as it is at the confluence of political, religious, economic and international tensions simmering in the Persian Gulf.  A majority Shi’a Muslim country ruled for hundreds of years by Sunni tribal chieftains with family ties to Saudi Arabia, Bahrain is a dictatorship whose people have regularly demanded political reform and seen their aspirations crushed.

Today, with strong support from the oil-rich Saudis, the Kingdom’s hard-line Al-Khalifa regime enjoys absolute powers, although the day-to-day political reality is often complex.

Bahrain is also the home of the U.S. Navy’s Fifth Fleet, which patrols those volatile waters, largely to keep an eye on Iran. The U.S. considers the port a critical element of its military posture in the Gulf. This consideration drives U.S. policy toward Bahrain. The centuries-old Bahraini-Saudi connection has, predictably, deterred the U.S. and other democratic countries from applying significant pressure to the kingdoms’ rulers. Stability is the byword.

Creating A Valid Process For Using Teacher Value-Added Measures

** Reprinted here in the Washington Post

Our guest author today is Douglas N. Harris, associate professor of economics and University Endowed Chair in Public Education at Tulane University in New Orleans. His latest book, Value-Added Measures in Education, provides an excellent, accessible review of the technical and practical issues surrounding these models. 

In particular, how should “value-added” measures be used in teacher evaluation? President Obama’s Race to the Top initiative expanded the use of these measures, which attempt to identify how much each teacher contributes to student test scores. In doing so, the initiative embraced and expanded the controversial reliance on standardized tests that started under President Bush’s No Child Left Behind.

In many respects, The Race was well designed. It addresses an important problem - the vast majority of teachers report receiving limited quality feedback on instruction. As a competitive grants program, it was voluntary for states to participate (though involuntary for many districts within those states). The Administration also smartly embraced the idea of multiple measures of teacher performance.

But they also made one decision that I think was a mistake.  They encouraged—or required, depending on your vantage point—states to lump value-added or other growth model estimates together with other measures. The raging debate since then has been over what percentage of teachers’ final ratings should be given to value-added versus the other measures. I believe there is a better way to approach this issue, one that focuses on teacher evaluations not as a measure, but rather as a process.