Shanker Blog

Premises, Presentation And Predetermination In The Gates MET Study

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

The National Education Policy Center today released a scathing review of last month’s preliminary report from the Gates Foundation-funded Measures of Effective Teaching (MET) project. The critique was written by Jesse Rothstein, a highly-respected Berkeley economist and author of an elegant and oft-cited paper demonstrating how non-random classroom assignment biases value-added estimates (also see the follow-up analysis).

Very quickly on the project: Over two school years (this year and last), MET researchers, working  in six large districts—Charlotte-Mecklenburg, Dallas, Denver, Hillsborough County (FL), Memphis, and New York City—have been gathering an unprecedented collection of data on teachers and students, grades 4-8.  Using a variety of assessments, videotapes of classroom instruction, and surveys (student surveys are featured in the preliminary report), the project is attempting to address some of the heretofore under-addressed issues in the measurement of teacher quality (especially non-random classroom assignment and how different classroom practices lead to different outcomes, neither of which are part of this preliminary report). The end goal is to use the information to guide in the creation of more effective teacher evaluation systems that incorporate high-quality multiple measures.

Despite my disagreements with some of the Gates Foundation’s core views about school reform, I think that they deserve a lot of credit for this project. It is heavily-resourced, the research team is top-notch, and the issues they’re looking at are huge.  The study is very, very important — done correctly. 

But Rothstein’s general conclusion about the initial MET report is that the results “do not support the conclusions drawn from them." Very early in the review, the following assertion also jumps off the page: "there are troubling indications that the Project’s conclusions were predetermined."

PISA For Our Time: A Balanced Look

Press coverage of the latest PISA results over the past two months has almost been enough to make one want to crawl under the bed and hide. Over and over, we’ve been told that this is a “Sputnik moment," that the U.S. among the lowest performing nations in the world, and that we’re getting worse.

Thankfully, these claims are largely misleading. Insofar as we’re sure to hear them repeated often over the next few years—at least until the next set of international results come in — it makes sense to try to correct the record (also see here and here).

But, first, I want to make it very clear that U.S. PISA results are not good enough by any stretch of the imagination, and we can and should do a whole lot better. Nevertheless, international comparisons of any kind are very difficult, and if we don’t pay careful attention to what the data are really telling us, it will be more difficult to figure out how to respond appropriately.

This brings me to three basic points about the 2009 PISA results that we need to bear in mind.

Death Of A Teacher Union Icon

The New Year brings sad word of the passing of Szeto Wah, celebrated Hong Kong democracy activist, legislator, and teacher union leader. He died on January 2 at the age of 79.

Once recognized by Time Magazine as one of the 25 most influential people in Hong Kong, and known by millions as "Uncle Wah," Szeto came to prominence in the 1970s as the firebrand founder of the Hong Kong Professional Teachers Union (PTU), which he led from 1974 to 1990. He was also a founder and leader of the Hong Kong Democratic Party, served in the Hong Kong legislature from 1985 to 2004, and was the founder and chairman of the Hong Kong Alliance in support of Patriotic Democratic Movements of China. The alliance was the leading organization offering support to the pro-democracy movement in Mainland China, which organized yearly protests on the anniversary of the Tiananmen Square massacre.

While condolences flow in from all over the world, the political question of the day in Hong Kong is whether or not the Chinese authorities will allow exiled democracy activists back into Hong Kong to attend Szeto’s funeral. Wang Dan, one of the most prominent of the Tiananmen Square democracy leaders, said that, for him, the loss is personal: "Uncle Wah has always been my personal mentor and a leader in the democratic movement. The greatest achievement he has made has been to pass on his beliefs before he left us. The younger generation now remembers June 4," he said.

We at the Shanker Institute also feel this as a personal loss. We met Szeto in 2002, when he travelled to Washington D.C. to deliver the Institute’s Albert Shanker Lecture. In it, he credited Al Shanker with helping to shape his political and organizational perspective:

The Year In Research On Market-Based Education Reform

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.

Race to the Top and Waiting for Superman made 2010 a banner year for the market-based education reforms that dominate our national discourse. By contrast, a look at the “year in research” presents a rather different picture for the three pillars of this paradigm: merit pay, charter schools, and using value-added estimates in high-stakes decisions.

There will always be exceptions (especially given the sheer volume of reports generated by think tanks, academics, and other players), and one year does not a body of research make.  But a quick review of high-quality studies from independent, reputable researchers shows that 2010 was not a particularly good year for these policies.

The 5-10 Percent Solution

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post.

In the world of education policy, the following assertion has become ubiquitous: If we just fire the bottom 5-10 percent of teachers, our test scores will be at the level of the highest-performing nations, such as Finland. Michelle Rhee likes to make this claim. So does Bill Gates.

The source and sole support for this claim is a calculation by economist Eric Hanushek, which he sketches out roughly in a chapter of the edited volume Creating a New Teaching Profession (published by the Urban Institute). The chapter is called "Teacher Deselection" (“deselection” is a polite way of saying “firing”). Hanushek is a respected economist, who has been researching education for over 30 years. He is willing to say some of the things that many other market-based reformers also believe, and say privately, but won’t always admit to in public.

So, would systematically firing large proportions of teachers every year based solely on their students’ test scores improve overall scores over time? Of course it would, at least to some degree. When you repeatedly select (or, in this case, deselect) on a measurable variable, even when the measurement is imperfect, you can usually change that outcome overall.

But anyone who says that firing the bottom 5-10 percent of teachers is all we have to do to boost our scores to Finland-like levels is selling magic beans—and not only because of cross-national poverty differences or the inherent limitations of most tests as valid measures of student learning (we’ll put these very real concerns aside for this post).

Do Americans Think We Spend Too Much On Education?

Cost-cutting is all the rage in education policy. This makes a lot of sense during a recession (the next few years will be brutal), and even during good times we all want money to be well-spent. But much of the discussion on this topic is less about weathering the storm than about a long-term effort to stop the growth of spending on public education. The underlying assumption, hardly unique to education policy, is that people are tired of increasing school costs, and want to start cutting back.

So, I wanted to take a quick look at what Americans think of education spending, now and over time, using data from the General Social Survey (1972-2008), a nationally representative sample of U.S. opinions and other characteristics (run by the National Opinion Research Center).  The question queries whether respondents believe the U.S. is spending too little, too much, or about the right amount on improving the nation’s education system (note the question’s use of "improving," which likely influences responses to some degree).  Also keep in mind that these are pre-recession data.

The 2008 data in the table below (non-missing sample size is 993) show that there’s actually a lot of agreement about education spending levels: Almost 3 in 4 Americans (71 percent) believe that we should spend more on improving education, while only about 1 in 20 feels that expenditures are too high.

Lyndon Johnson: Education Reformer

Underlying virtually all contemporary education policy debates is the question of poverty. Certainly, high poverty and inequality do not mean we shouldn’t improve schools. On the other hand, the standpoint of some in the debate today evolved from an inarguable, commendable notion (poor kids can learn too) to an ideological brick wall, behind which those who dare speak poverty’s name are accused of “making excuses."

Anyone who reads history (or who has lived through it) knows that the tension between poverty and equality of educational opportunity is nothing new, nor is the debate about how to address them.  For example, these same issues arose during the campaign to pass the Elementary and Secondary Education Act (ESEA) in 1965, as President Lyndon Johnson cajoled everyone he could to push the bill through Congress.

Unlike most of the political leaders who are the self-proclaimed education reformers of today, Johnson had been a teacher. His teaching experience, at a small segregated school for Mexican Americans in the impoverished town of Cotulla, Texas, convinced him that poverty and educational inequality must be tackled in tandem.

Full disclosure: I have a minor obsession with Lyndon Johnson (it feels good to say that out loud), and I have read all of the released transcripts of the phone calls and White House meetings that LBJ recorded. In one of these conversations—on March 6, 1965—Johnson is speaking with Hubert Humphrey, his newly-inaugurated Vice-President, who spent much of his term serving as LBJ’s liaison to Congress. Johnson’s deep belief in quality public education as a key to reducing poverty comes across in this conversation.

Why Does Joel Klein Keep Misrepresenting Al Shanker?

Outgoing New York City Chancellor Klein loves to try to wrap himself in the mantle of Al Shanker. He is especially fond of pulling clipped Shanker quotes out of his hat—and out of context—when speaking about his favorite education “reforms." At first this may seem puzzling, because the ex-Chancellor is disinclined to give either the United Federation of Teachers or its parent organization, the American Federation of Teachers, credit for much of anything except intransigence. It must be an inconvenient truth for Klein that Shanker devoted his life to making both organizations into the strong and aggressive advocates for teachers and teaching that they continue to be.

In "What I Learned at the Barricades," a December 6 Wall Street Journal column, Klein leads up to his latest Shanker references with a characteristic litany of inaccurate claims – ones that Al would be quick to correct:

First, it is wrong to assert that students’ poverty and family circumstances severely limit their educational potential." And “Second, traditional proposals for improving education—more money, better curriculum, smaller classes, etc —aren’t going to get the job done.

Really? It’s hard to imagine which barricades Klein learned at. There is plenty of evidence to support the impact of all of these.

But, for those of us who knew and worked closely with Al (I did from 1967-1984 and from 1989 until his death in 1997), what’s truly galling is Klein’s distorted use of Al’s thinking to shore up a simplistic, narrowly punitive agenda that Shanker would have discredited.

Whom Do Americans Trust? Nurses...And Teachers!

Just a couple months after the prestigious Institute of Medicine urged that nurses be seen as “full partners” in redesigning the American health care system, they have received another vote of confidence, this one from the American public. According to the most recent Gallup poll, for the 9th straight year (and the 11th year in all), the American people ranked nurses as the most honest and ethical workers in the country.

When asked to the rate the ethics and honesty of people in a variety of occupations, 81 percent of those surveyed gave a “very high/high” rating to nurses. Doctors received a very high/high rating from a still respectable of 66 percent of respondents.

Despite being regularly scapegoated by politicians and the media for the past several years, grade school teachers still edged out doctors by 1 percentage point (a statistical tie),  with 67 of those polled expressing high regard for the profession. Although this places teachers fairly high on the list of trusted professions—in fourth place, behind nurses (81 percent), military officers (73 percent), and pharmacists (71 percent)—the teacher bashing has apparently had an effect: Teachers have lost ground since the 2007 version of this survey, when they were rated "very high/high" by 74 percent of those surveyed.

The War On Error

The debate on the use of value-added models (VAM) in teacher evaluations has reached an impasse of sorts. Opponents of VAM use contend that the imprecision is too high for the measures to be used in evaluation; supporters argue that current systems are inadequate, that all measures entail error but this doesn’t preclude using the estimates. 

This back-and-forth may be missing the mark, and it is not particularly useful in the states and districts that are already moving ahead. The more salient issue, in my view, is less about the amount of error than about how it is dealt with when the estimates are used (along with other measures) in evaluation systems.

Teachers certainly understand that some level of imprecision is inherent in any evaluation method—indeed, many will tell you about colleagues who shouldn’t be in the classroom, but receive good evaluation ratings from principals year after year. Proponents of VAM often point to this tendency of current evaluation systems to give “false positive” ratings as a reason to push forward quickly. But moving so carelessly that we disregard the error in current VAM estimates—and possible methods to reduce its negative impacts—is no different than ignoring false positives in existing systems.