K-12 Education

Rethinking The Use Of Simple Achievement Gap Measures In School Accountability Systems

So-called achievement gaps – the differences in average test performance among student subgroups, usually defined in terms of ethnicity or income –  are important measures. They demonstrate persistent inequality of educational outcomes and economic opportunities between different members of our society.

So long as these gaps remain, it means that historically lower-performing subgroups (e.g., low-income students or ethnic minorities) are less likely to gain access to higher education, good jobs, and political voice. We should monitor these gaps; try to identify all the factors that affect them, for good and for ill; and endeavor to narrow them using every appropriate policy lever – both inside and outside of the educational system.

Achievement gaps have also, however, taken on a very different role over the past 10 or so years. The sizes of gaps, and extent of “gap closing," are routinely used by reporters and advocates to judge the performance of schools, school districts, and states. In addition, gaps and gap trends are employed directly in formal accountability systems (e.g., states’ school grading systems), in which they are conceptualized as performance measures.

Although simple measures of the magnitude of or changes in achievement gaps are potentially very useful in several different contexts, they are poor gauges of school performance, and shouldn’t be the basis for high-stakes rewards and punishments in any accountability system.

Multiple Measures And Singular Conclusions In A Twin City

A few weeks ago, the Minneapolis Star Tribune published teacher evaluation results for the district’s public school teachers in 2013-14. This decision generated a fair amount of controversy, but it’s worth noting that the Tribune, unlike the Los Angeles Times and New York City newspapers a few years ago, did not publish scores for individual teachers, only totals by school.

The data once again provide an opportunity to take a look at how results vary by student characteristics. This was indeed the focus of the Tribune’s story, which included the following headline: “Minneapolis’ worst teachers are in the poorest schools, data show." These types of conclusions, which simply take the results of new evaluations at face value, have characterized the discussion since the first new systems came online. Though understandable, they are also frustrating and a potential impediment to the policy process. At this early point, “the city’s teachers with the lowest evaluation ratings” is not the same thing as “the city’s worst teachers." Actually, as discussed in a previous post, the systematic variation in evaluation results by student characteristics, which the Tribune uses to draw conclusions about the distribution of the city’s “worst teachers," could just as easily be viewed as one of the many ways that one might assess the properties and even the validity of those results.

So, while there are no clear-cut "right" or "wrong" answers here, let’s take a quick look at the data and what they might tell us.

Librarians, Libraries, Serendipity And Passion

Our guest author today is Connie Williams, a National Board Certified Teacher librarian at Petaluma High School in Petaluma, CA, past president of the California School Library Association, and co-developer of the librarian and teacher 2.0 classroom tutorials.

Down the road from where I live, on the first-of-the month, a group of vintage car owners gather for a “cars and coffee” meet up. The cars that show up with their drivers cover many years and obsessions. Drivers park, open up the car hoods and take a few steps back and begin talking with other car owners and visitors who happen by. These are people who are interested in the way cars work, their history, and they all have stories to share.

How do they know so much about their cars? They work on them – gaining insight by hands-on practice and consultations with experts. If they’re wealthy enough, they pay someone else to do the work, yet they don’t just hand over their cars to them. They read about them, participate in on-line groups, ask for guidance, and they drive them. Most often, when they drive them, someone stops and asks questions about their cars and they teach what they know to others. 

This is an example of the kind of learning we would hope for, for all our students – a passion that is ignited and turns into knowledge that is grown, developed, and shared. In this sense, it is inquiry – asking questions and taking the required steps to answer them – that is at the heart of learning.

The Bewildering Arguments Underlying Florida's Fight Over ELL Test Scores

The State of Florida is currently engaged in a policy tussle of sorts with the U.S. Department of Education (USED) over Florida’s accountability system. To make a long story short, last spring, Florida passed a law saying that the test scores of English language learners (ELLs) would only count toward schools’ accountability grades (and teacher evaluations) once the ELL students had been in the system for at least two years. This runs up against federal law, which requires that ELLs’ scores be counted after only one year, and USED has indicated that it’s not willing to budge on this requirement. In response, Florida is considering legal action.

This conflict might seem incredibly inane (unless you’re in one of the affected schools, of course). Beneath the surface, though, this is actually kind of an amazing story.

Put simply, Florida’s argument against USED's policy of counting ELL scores after just one year is a perfect example of the reason why most of the state's core accountability measures (not to mention those of NCLB as a whole) are so inappropriate: Because they judge schools’ performance based largely on where their students’ scores end up without paying any attention to where they start out.

The Equity Projection

A new Mathematica report examines the test-based impact of The Equity Project (TEP), a New York City charter school serving grades 5-8. TEP opened up for the 2009-10 school year, receiving national attention mostly due to one unusual policy: They paid teachers $125,000 per year, regardless of experience and education, in addition to annual bonuses (up to $25,000) for returning teachers. TEP largely makes up for these unusually high salary costs by minimizing the number of administrators and maintaining larger class sizes.

As is typical of Mathematica, the TEP analysis is thorough and well-done. The school's students' performance is compared to that of similar peers with a comparable probability of enrolling in TEP, as identified with propensity scores. In general, the study’s results were quite positive. Although there were statistically discernible negative impacts of attendance for TEP’s first cohort of students during their first two years, the cumulative estimated test-based impact was significant, positive and educationally meaningful after three and four years of attendance. As always, the estimated effect was stronger in math than in reading (estimated effect sizes for the former were very large in magnitude). The Mathematica researchers also present analyses on student attrition, which did not appear to bias the estimates substantially, and they also show that their primary results are robust when using alternative specifications (e.g., different matching techniques, score transformations, etc.).

Now we get to the tricky questions about these results: What caused them and what can be learned as a result? That’s the big issue with charter analyses in general (and with research on many other interventions): One can almost never separate the “why” from the “what” with any degree of confidence. And TEP, with its "flagship policy" of high teacher salaries, which might appeal to all "sides" in the education policy debate, provides an interesting example in this respect.

Not All Discipline Disparities May Be The Result Of Implicit Bias

Over the past few months, we have heard a lot about discipline disparities by race/ethnicity and gender -- disparities that begin in the earliest years of schooling. According to the Civil Rights Data Collection Project by the U.S. Department of Education's Office for Civil Rights, "black students represent 18% of preschool enrollment, but 42% of preschool students suspended once and 48% of students suspended more than once." It also found that "boys receive more than three out of four out-of-school preschool suspensions."

This focus on student discipline disparities has also drawn attention to the research on implicit bias -- the idea that we all harbor unconscious attitudes that tend to favor individuals from some groups (whites, males, those judged to be good looking, etc.), and that disadvantage people from other groups (people of color, women, ethnic minorities, etc.). The concept of implicit bias suggests that good or bad behavior is often in the eye of the beholder, and disparities in disciplinary outcomes (e.g., suspensions and expulsions) may be influenced by unconscious stereotypes.

Part of me is very glad that we are finally having this conversation. Acknowledging the existence and consequences of subtle, implicit forms of prejudice is an important and necessary first step toward mitigating their effects and advancing toward fairness -- see my implicit bias series here. But it sometimes seems that the discipline and the implicit bias conversations are one and the same, and this concerns me for two reasons.

Where Al Shanker Stood: Democracy In Public Education

There is an ongoing debate in the U.S. about the role of democracy in public education. In March 1997, in his final “Where We Stand” column in the New York Times, Al Shanker addressed this issue directly. The piece, published posthumously, was an excerpt from a larger essay entitled "40 Years in the Profession," which was included in a collection published by the Phi Delta Kappa Educational Foundation.

Why do I continue when so much of what I’ve worked for seems threatened? To a large extent, because I believe that public education is the glue that has held this country together. Critics now say that the common school never really existed, that it’s time to abandon this ideal in favor of schools that are designed to appeal to groups based on ethnicity, race, religion, class, or common interests of various kinds. But schools like these would foster divisions in our society; they would be like setting a time bomb.

A Martian who happened to be visiting Earth soon after the United States was founded would not have given this country much chance of surviving. He would have predicted that this new nation, whose inhabitants were of different races, who spoke different languages, and who followed different religions, wouldn’t remain one nation for long. They would end up fighting and killing each other. Then, what was left of each group would set up its own country, just as has happened many other times and in many other places. But that didn’t happen. Instead, we became a wealthy and powerful nation—the freest the world has ever known. Millions of people from around the world have risked their lives to come here, and they continue to do so today.