At a press conference earlier this week, New York City Mayor Michael Bloomberg announced the city’s 2011 test results. Wall Street Journal reporter Lisa Fleisher, who was on the scene, tweeted Mayor Bloomberg’s remarks. According to Fleisher, the mayor claimed that there was a “dramatic difference” between his city’s testing progress between 2010 and 2011, as compared with the rest of state.

Putting aside the fact that the results do not measure “progress” per se, but rather cohort changes – a comparison of cross-sectional data that measures the aggregate performance of two different groups of students – I must say that I was a little astounded by this claim. Fleisher was also kind enough to tweet a photograph that the mayor put on the screen in order to illustrate the “dramatic difference” between the gains of NYC students relative to their non-NYC counterparts across the state.  Here it is:

Dramatic? In short, the graph shows that the overall math proficiency of NYC students increased 3.3 percentage points, compared with a 1.8 point increase in the rest of the state; the NYC reading rate went up 1.5 percentage points, compared with a 1.2 point decrease throughout the rest of New York. So, the change in NYC vis-à-vis the rest of the state was 1.5 percentage points higher in math, and 2.7 points higher in reading (about half of that due to the fact that the state’s reading pass rate declined a bit). That’s it.

Given all the measurement error in standardized assessments, the (related) imprecision in determining the “cut scores” above which students are deemed “proficient," and the cross-sectional nature of these data, the safest interpretation of the NYC results (and those for the whole state) is that there has been little or no change, positive or negative. Because the sample (number of students taking the test) is so large, the NYC score changes are probably statistically discernible (though it's tougher to say how the distributions play out when the scores are used to derive proficiency rates). But the cohort-to-cohort increases are not large enough to be particularly meaningful in educational terms, whether by themselves or compared with the changes in the rest of the state.

I suppose that everyone is entitled to their own definition of “dramatic," but, with all due respect to the mayor and his team, it’s not that easy to see what all the excitement is about.

- Matt Di Carlo