Skip to:

If Your Evidence Is Changes In Proficiency Rates, You Probably Don't Have Much Evidence


The recent article by the Atlanta Journal Constitution on the disparities of test scores and the possible link to cheating is similar to this. Without a robust statistical model and method, any change is wide open to interpretation, and we should have coherent policies in place.

I think you are taking the wrong approach here. You should go at the assumptions, to examine when/why they might be valid. One assumption is that these proficiency rates are meaningful, in and of themselves. The problem is that they are set rather arbitraryily -- even if not capriously. Changes in proficiency rates could radically alter with different cut scores for proficiency. On the other hand, the more confidence we have in the setting of the proficiency cut scores (i.e. the validity of the cut scores), the more confidence we can have in changes in those proficiency rate. Unfortunately, there is little reason to have confidence in them today, and therefore little reason to have confidence in changes in proficiency rates. Problem. Implication. Solution. Another assumption is that changes in rates can applied on the state, district, school and perhaps even grades and teacher levels. This gets to a sample size issue. Comparisons on the state level might actually be meaningful. In large districts, they might be meaningful. But when looking a particular subgroups -- especially when they are small subgroups -- it gets much more problematic. Confidence intervals and margins of error are well known statistical techniques for addressing this issue. Unfortunately, most changes we see reported do NOT fall outside the margins of error/confidence intervals. While it may be easier to ignore that fact, we can't let convenience overwhelm what we actually know. When it comes to multi-variate analysis of longitudinal test data, there are other problems. I think it would be good if you wrong a companion piece that was as critical of that approach as you are generally are of this one.


This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the Shanker Blog may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.