What The "No Excuses" Model Really Teaches Us About Education Reform
** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post
In a previous post, I discussed “Apollo 20," a Houston pilot program in which a group of low-performing regular public schools are implementing the so-called “no excuses” education model common among high-profile charter schools such as KIPP. In the Houston implementation, “no excuses” consists of five basic policies: a longer day and year, resulting in 21 percent more school time; different human capital policies, including performance bonuses and firing and selectively rehiring all principals and half of teachers (the latter is one of the "turnaround models" being pushed by the Obama Administration); extensive 2-on-1 tutoring; regular assessments and data analysis; and “high expectations” for behavior and achievement, including parental contracts.
A couple of weeks ago, Harvard professor Roland Fryer, the lead project researcher, released the results of the pilot’s first year. I haven’t seen much national coverage of the report, but I’ve seen a few people characterize the results as evidence that “’No excuses’ works in regular public schools." Now, it’s true that there were effects – strong in math – and that the results appear to be persistent across different model specifications.
But, when it comes to the question of whether “no excuses works," the reality is a bit more complicated. There are four main things to keep in mind when interpreting the results of this paper, a couple of which bear on the larger debate about "no excuses" charter schools and education reform in general.
First, and most obviously, these results pertain only to the program’s first year. That is not enough time to draw anything beyond extremely tentative conclusions about whether a given intervention “works." On a related note, it bears mentioning that even the final results will still leave open important common questions such as selection (the evaluation is not experimental) and scalability.
Second, results were basically mixed. The program’s effects were rather strong in math - on average, about 0.25 standard deviations, a substantial effect size, with the magnitude and statistical significance of the estimates varying widely by grade, school and other variables (for example, treatment effects were only marginally significant in middle and most high school grades). But the reading effects were essentially nil. While there was again some variation by grade and other subgroups, the average reading effect was quite small (about 0.05 standard deviations), and only very marginally statistically significant, which means that we can’t have much faith in its precision. So, to whatever small degree this analysis provides tentative evidence as to whether the “no excuses” model “worked," the evidence is somewhat inconsistent. As is frequently the case, the education policy interventions in fashion these days (especially charter schools), when they work, tend to have meaningful effects on only one of the two commonly-tested subjects. This is in no small part because the factors, such as content knowledge, that determine reading ability are developed before and outside of schools.
The third thing to keep in mind, which I discussed in my prior post about Apollo 20, is the fact that this program – and, by extension, the “no excuses” model in general – is quite expensive. Fryer reports a cost of around $2,000 per pupil (the total annual cost of the program is $19 million), with amounts varying by whether or not students received the tutoring. The overall additional spending represents a 20 percent increase over average per-pupil expenditures. Fryer notes that this dollar amount is similar to the spending of other “no excuses” charters, such as those in New York City (though he provides no citation for his figures on other charters, which are difficult to track [and compare across locations]).
Nevertheless, let’s just say his numbers are correct, and let’s also take a huge leap and say that the achievement return on this spending is worthwhile (as Fryer argues is the case for the math results, using a very rough calculation). This is a massive investment, even if one takes steps to lower the costs (e.g., by using small group instead of 2-on-1 tutoring). The Apollo 20 program requires a roughly 20 percent increase in spending during a time when many states, including California, are actually cutting spending by similar proportions. So, to the extent that the eventual final Apollo 20 results suggest that the “no excuses” model might be expanded into regular public schools (and that’s still an open question), it will require a huge cash infusion.
My fourth point about this paper is a more general conceptual one, which pertains to the “no excuses” philosophy. As I have discussed several times, the best question to ask is not whether “no excuses” (and charter schools in general) works, but why. And this paper only helps a little bit. It gives us five policies to consider, but doesn’t really isolate the unique effects of each of the five policies implemented (for example, by varying them between schools).
There is one partial exception – tutoring. The math gains among Apollo 20 students were in no small part driven by progress in grades (six and nine) that received the 2-on-1 tutoring. Estimated effects in the non-tutored grades were much more moderate, and there was no tutoring in reading, which was largely unaffected. In addition, Fryer estimates directly the effect of the tutoring treatment (math only), which is large and statistically significant. So, this represents decent circumstantial evidence that tutoring is a huge factor in the math results. I have previously speculated that the extended time is also a big one, but I can’t be sure.
In any case, among these five interventions (tutoring, extended time, improving human capital, interim assessments and “high expectations”), only one of them – “improving human capital” through more selective hiring and performance bonuses – focuses directly on improving teacher quality, the primary tool advocated by market-based reformers. Frankly, the human capital component is really the only one that could be called “market-based” by any reasonable definition (though the regular analysis of interim assessment data might be loosely classified as such).
In other words, the teacher-focused, market-based philosophy that dominates our public debate is not very well represented in the “no excuses” model, even though the latter is frequently held up as evidence supporting the former. Now, it’s certainly true that policies are most effective when you have good people implementing them, and that the impact of teachers and administrators permeates every facet of schools’ operation and culture. Nonetheless, most of the components that comprise the “no excuses” model in its actual policy manifestation are less focused on “doing things better” than on doing them more. They’re about more time in school, more instructional staff, more money and more testing. I’ve called this a “blunt force” approach to education, and that’s really what it is. It’s not particularly innovative, and it’s certainly not cheap.
To be clear, I’m obviously not saying that “no excuses” is somehow inconsistent with teacher-focused or market-based reform. Those who consider themselves “reformers” are a diverse group, and they embrace a variety of different approaches.
What I am saying is that, to whatever extent “no excuses” works in regular public schools (and, again, that remains a completely open question), it implies that we might consider focusing a bit more on the conditions and policies of schools, in addition to the people that work in them. And it also suggests that we should stop fooling ourselves into thinking that we can make drastic cuts to education funding and get better results.
- Matt Di Carlo
"It gives us five policies to consider, but doesn’t really isolate the unique effects of each of the five policies implemented (for example, by varying them between schools)."
So what? Could this be like asking which of the five ingredients in a cake is the "real" cake (flour, sugar, eggs, butter, water)?
It is difficult to determine whether a program works because it is still based on the artificial test. The test measures winning but not learning.
"No excuses" is just more Rhetoric. Kids learn in different ways, demonstrate learning in different ways and blossom at different times due to a whole bunch of reasons. When we quit treating them like the Stepford Kids and demanding educational purity, then and only then will we succeed.
At Lee High School, celebrated as the most successful Apollo school, scores on the math TAKS did go up. But the way those gains were achieved were by math tutors and teachers sacrificing the math curriculum to drill students on guess-and-check test taking methods. What will those students do if asked to solve a math question that doesn't come with 4 answers, one that is guaranteed to be correct?
All the other tenets - firing teachers, extended hours that wasted on excessive testing - produced no results in reading. So the only benefit from Apollo was the tutoring. Everything else cancelled each other out. That's using Fryer's own metrics. (By the way, those should be considered distorted at best and invalid at worst, because Grier and Fryer made it very clear to principals that their jobs were on the line if those particular numbers didn't come out looking good.)
Worse, if you consider other metrics, Apollo actually harmed education in Houston. At Lee High School, the number of AP exams on which students earned 3's or higher fell by nearly half:
At the flagship Apollo school, tutoring helped students pass a low-level math test. The other four tenets were implemented so badly as to actually harm education at the school. If that's the best of the Apollo schools, then how much worse was Apollo elsewhere?
It should be noted that while Fryer has said he identified the five characteristics of successful charter schools in 2009, he hasn't published or issued a working paper with that research. Everything available starts with different research questions and assumes the five characteristics.
The most important evidence here is the numbers, percentages, and types of students who did not stick out the No Excuses rigor. The district reported in January 2011 that 7,385 students were enrolled, with 86.6% being economically disadvantaged. On page 71, Fryer reports that the tested sample was 61% economically disadvantaged, and there were 8,693 "observations." Do you know what that term means? Should it be read as 8,693 tests, and two tests were given, mean that 1/2 of them stuck it out to the end? Or does it mean something completely different indicating that Fryer did not address the attrition number? Either way, Fryer writes clearly, at the beginning of the paper, that the program had about 7,000 students. Why did he not write a second, clear prose sentence saying how many were tested in the spring?
Do you know how many teachers left SHS by mid-year last year. Entire hallways of teachers will not be returning. Good teachers. 30 to 40 percent of the kids act like animal, holding fellow students and teachers hostage. A WAR zone. Sickening. The system is broken.