• In Research, What Does A "Significant Effect" Mean?

    If you follow education research – or quantitative work in any field – you’ll often hear the term “significant effect." For example, you will frequently read research papers saying that a given intervention, such as charter school attendance or participation in a tutoring program, had “significant effects," positive or negative, on achievement outcomes.

    This term by itself is usually sufficient to get people who support the policy in question extremely excited, and to compel them to announce boldly that their policy “works." They’re often overinterpreting the results, but there’s a good reason for this. The problem is that “significant effect” is a statistical term, and it doesn’t always mean what it appears to mean. As most people understand the words, “significant effects” are often neither significant nor necessarily effects.

    Let’s very quickly clear this up, one word at a time, working backwards.

  • NAEP Shifting

    ** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

    Tomorrow, the education world will get the results of the 2011 National Assessment of Educational Progress (NAEP), often referred to as the “nation’s report card." The findings – reading and math scores among a representative sample of fourth and eighth graders - will drive at least part of the debate for the next two years, when the next round comes out.

    I’m going to make a prediction, one that is definitely a generalization, but is hardly uncommon in policy debates: People on all “sides” will interpret the results favorably no matter how they turn out.

    If NAEP scores are positive – i.e., overall scores rise by a statistically significant margin, and/or there are encouraging increases among key subgroups such as low performers or low-income students – supporters of market-based reform will say that their preferred policies are working. They’ll claim that the era of test-based accountability, which began with the enactment of No Child Left Behind ten years ago, have produced real results. Market reform skeptics, on the other hand, will say that virtually none of the policies, such as test-based teacher evaluations and merit pay, for which reformers are pushing were in force in more than a handful of locations between 2009 and 2011. Therefore, they’ll claim, the NAEP progress shows that the system is working without these changes.

    If the NAEP results are not encouraging – i.e., overall progress is flat (or negative), and there are no strong gains among key subgroups – the market-based crowd will use the occasion to argue that the “status quo” isn’t producing results, and they will strengthen their call for policies like new evaluations and merit pay. Skeptics, in contrast, will claim that NCLB and standardized test-based accountability were failures from the get-go. Some will even use the NAEP results to advocate for the wholesale elimination of standardized testing.

  • Similar Problems, Different Response: “We Are Public Education”

    Thousands of people from all over Spain demonstrated Saturday October 22nd in Madrid against severe austerity measures affecting public education in several Spanish regions. The march on Madrid, which attracted more than 100,000 protesters – huge by Spanish standards – was jointly organized by national education unions and the national parents’ association, CEAPA. Taking part in the protest, a somewhat unprecedented coalition: educators, parents, and students.

    The economy in Spain is in terrible shape. Parents and teachers don’t always have an ideal relationship, yet  Spaniards seem to have avoided the divisive and unproductive quarrels we often read about in the US education debate – e.g., adults versus children or teachers versus parents – in an attempt to prioritize long-term educational investment over short-term, budget-driven savings. This broad alliance is building consensus around the notion of “the education community." As the protest’s manifesto notes, such community is “society as a whole," which must unite to oppose drastic budget cuts in public education and attacks by political leaders on public school teachers.

    The nationwide protest was triggered by a recent government decision that bans the temporary hiring of teachers as part of a plan to reduce government spending. In various parts of the country, teachers have already been laid off, class sizes and teaching hours have increased significantly, and teachers will have to teach subjects they are not specialized in. Many schools will have to reduce extra-curricular activities, remedial classes for struggling students and integration classes for the children of immigrants. This situation triggered a series of regional demonstrations across Spain throughout the months of September and October – including student demonstrations in defense of public education – with protesters arguing that education quality has been put at risk. National in scale, the march on Madrid sends a broader message, with the potential of immediate political impact.

  • The Ratings Game: New York City Edition

    Gotham Schools reports that the New York City Department of Education rolled out this year’s school report card grades by highlighting the grades’ stability between this year and last. That is, they argued that schools’ grades were roughly the same between years, which is supposed to serve as evidence of the system’s quality.

    The city’s logic here is generally sound. As I’ve noted before, most schools don’t undergo drastic changes in their operations over the course of a year, and so fluctuations in grades among a large number of schools might serve as a warning sign that there’s something wrong with the measures being used. Conversely, it’s not unreasonable to expect from a high-quality rating system that, over a two-year period, some schools would get higher grades and some lower, but that most would stay put. That was the city’s argument this year.

    The only problem is that this wasn’t really the case.

  • Higher Education: The Great Equalizer Or Business As Usual?

    ** Also posted here on "Valerie Strauss' Answer Sheet" in the Washington Post

    Several weeks ago, a survey of college admission directors and enrollment managers conducted by Inside Higher Education sparked considerable media coverage about an issue that is not entirely new: Money, not only merit, matters in college admissions.

    According to the survey of 462 directors and managers, in the face of generalized budget cuts, universities are favoring applicants who don't need financial assistance to pay their tuition. About  22 percent agreed that “the financial downturn [had] forced them to pay more attention to an applicants’ ability to pay when [making] admissions decisions." Directors acknowledged seeking more candidates who would not need financial aid, including out-of-state and international students. Furthermore, 10 percent of four-year colleges reported that the admitted students who could pay in full had lower grades than their peers who couldn’t.

    These findings resulted in headlines like “College Admission Directors Finally Admit They Want Rich Students More Than Smart Students” or “Universities Seeking Out Students of Means." Journalists wrote that higher education institutions are no longer in the business of recruiting “the best and brightest […] but the richest” and noted that “money is talking a bit louder in college admissions these days.” David A. Hawkins, director at the National Association for College Admission Counseling, told the New York Times: “As institutional pressures mount, between the decreased state funding, the pressure to raise a college’s profile, and the pressure to admit certain students, we’re seeing a fundamental change in the admissions process. Where many of the older admissions professionals came in through the institution and saw it as an ethically centered counseling role, there’s now a different dynamic that places a lot more emphasis on marketing."

    The actual finding that money matters in admissions did not strike me as a particularly surprising. After all, social scientists have long been skeptical of meritocracy’s role in higher education – see here and here. What surprised me was that survey respondents were comfortable giving a “socially inappropriate” answer. Let me explain.

  • Do Americans Think Government Should Reduce Income Inequality?

    With all the recent coverage of Occupy Wall Street and President Obama’s jobs bill, we’ve heard a lot of polling results showing that a large plurality of Americans supports raising taxes on high earners, and that this support is strong among both Democrats and Republicans.

    The campaign to raise taxes on high-income households is part of a larger ideological notion that reducing inequality by such means as taxation and welfare programs is a proper function of government. Supporters (e.g., Democrats) argue that progressive taxation helps to ensure that high earners pay their “fair share” in supporting the public resources, such as schools, roads and law enforcement, that are necessary (but not sufficient) for their success. Republicans, on the other hand, tend frame the issue directly in terms of government intrusion – the government is unfairly “picking winners and losers," and stifling innovation and risk-taking. The assumption seems to be that many Americans don't care for the generic idea of government taking an active role in reducing the gap between rich and poor, even though they tend to support many of the specific means by which this occurs, including not only raising taxes on high earners, but also public education and programs like Medicaid.

    So, it might be interesting to see what Americans think of the broader idea that government has a legitimate role in reducing income inequality. Let’s take a quick look.

  • The Teachers' Union Hypothesis

    For the past couple of months, Steve Brill's new book has served to step up the eternally-beneath-the-surface hypothesis that teachers’ unions are the primary obstacle to improving educational outcomes in the U.S. The general idea is that unions block “needed reforms," such as merit pay and other forms of test-based accountability for teachers, and that they “protect bad teachers” from being fired.

    Teachers’ unions are a convenient target. For one thing, a significant proportion of Americans aren’t crazy about unions of any type. Moreover, portraying unions as the villain in the education reform drama facilitates the (mostly false) policy-based distinction between teachers and the organizations that represent them – put simply, “love teachers, hate their unions." Under the auspices of this dichotomy, people can advocate for changes , such as teacher-level personnel policies based partially on testing results, without having to address why most teachers oppose them (a badly needed conversation).

    No, teachers’ unions aren’t perfect, because the teachers to whom they give voice aren’t perfect. There are literally thousands of unions, and, just like districts, legislatures and all other institutions, they make mistakes. But I believe strongly in separating opinion and anecdote from actual evidence, and the simple fact is that the pervasive argument that unions are a substantial cause of low student performance has a weak empirical basis, while the evidence that unions are a primary cause of low performance does not exist.

  • Trouble In Paradise

    According to the principles of market-based education reform, there’s at least one large, urban public school district operating at max power: District of Columbia Public Schools.

    For the past 2-3 years, DCPS has been a reformer’s paradise. The district has a new evaluation system (IMPACT), which it designed by itself. The system includes heavily-weighted value-added estimates (50 percent for teachers in tested grades/subjects), and the results of teachers’ evaluations are used every year to fire the teachers who receive the lowest evaluation ratings, or receive the second lowest score for two consecutive years. “Ineffective teachers” are being weeded out – no hearing, no due process, no nothing.

    Furthermore, these evaluation scores are also used to award performance bonuses, and very large ones at that – up to $25,000. This should, so the logic goes, be attracting high-achieving people to DCPS, and keeping them around after they arrive. And, finally, as a result of many years of growth, the city has among the largest charter school sectors in the nation, with almost half of public school student attending charters. Theoretically, this competition should be upping the game of all schools, charter and regular public alike.

    Basically, almost everything that market-based reformers think needs to happen has been the reality in DCPS for the past 2-3 years. And the staff  has been transformed too. The majority of principals, and a huge proportion of teachers, were hired during the tenure of either Michelle Rhee or her successor, Kaya Henderson.

    The district should be in overdrive right about now. Is it?

  • What The "No Excuses" Model Really Teaches Us About Education Reform

    ** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

    In a previous post, I discussed “Apollo 20," a Houston pilot program in which a group of low-performing regular public schools are implementing the so-called “no excuses” education model common among high-profile charter schools such as KIPP. In the Houston implementation, “no excuses” consists of five basic policies: a longer day and year, resulting in 21 percent more school time; different human capital policies, including performance bonuses and firing and selectively rehiring all principals and half of teachers (the latter is one of the "turnaround models" being pushed by the Obama Administration); extensive 2-on-1 tutoring; regular assessments and data analysis; and “high expectations” for behavior and achievement, including parental contracts.

    A couple of weeks ago, Harvard professor Roland Fryer, the lead project researcher, released the results of the pilot’s first year. I haven’t seen much national coverage of the report, but I’ve seen a few people characterize the results as evidence that “’No excuses’ works in regular public schools." Now, it’s true that there were effects – strong in math – and that the results appear to be persistent across different model specifications.

    But, when it comes to the question of whether “no excuses works," the reality is a bit more complicated. There are four main things to keep in mind when interpreting the results of this paper, a couple of which bear on the larger debate about "no excuses" charter schools and education reform in general.

  • An International Perspective On Corporate Pay

    Our guest author today is Michael Tims, associate professor of biology at Montgomery College in Takoma Park, Maryland. Some of his writing can be found on his science blog, Bardo's Calculus, as well as at the Hyattstown Mill Arts Project, where he is a board member.

    The growing wealth gap in the United States has worried some commentators for years. The length and breadth of the economic crisis, and the suffering it has brought with it, have moved those concerns into the mainstream. One aspect of this development that warrants more attention is the connection between declining rates of unionization, and the incredible gap between the pay of workers and their bosses.

    As corporate resistance to unions has increased and union density declined, the discrepancy in pay between management and worker has grown extreme. Since the mid 1970s, the average multiple of CEO pay to worker pay has increased from 28x in 1970 to 158x in 2005, to almost 400x in 2010. . Their average "total realized annual CEO compensation" is currently $12 million, according to Governance Metrics International. During this same period, worker pay has stagnated and fallen behind inflation, despite an historic rise in workforce productivity

    This phenomenon of high pay disparity in the industrial world is uniquely American, with the next highest countries being Britain (25x), Sweden (13x), Germany (11x) and Japan (10x). Claims that these pay levels represent success on the part of the CEO appear to be misleading.