Skip to:

The Thrill Of Success, The Agony Of Measurement


Is there any data about how many students who start in K end up staying, and completing, 1st, 2nd, 3rd, 4th, grade and so on. What happens if 50 students start in K and then a bunch leave by the time the 3rd graders take the test? (I'm not saying this is happening here. I'm just curious if it is.) How does the above chart take into account persistence rates? I think some call this the cohort. Anyone?

So would a Success lottery study, like Kane/Angrist did in Boston, be convincing?

Mike, Convincing of what? MD

Effectiveness. I.e., if your concern is about using absolute scores to gauge efficacy (Agreed!), would that particular study design answer your questions?

Mike, I'm still slightly confused (perhaps my fault). If you're asking me whether a lottery study would assuage my concerns about the use of absolute measures to gauge school effectiveness, the answer is no - it would/could not (but I don't think that's your question). If, on the other hand, you're asking whether a lottery study would convince me that Success Academies are relatively effective in boosting test scores, I would say, as I tried to say in the post (paragraph starting "To be clear..."), that I'm not particularly skeptical of that conclusion, only about reaching it using proficiency rates. Thanks for your comment, MD

Educator, In the chart, each row represents a single cohort (third graders in each school in 2014). So, if, for example, there were students in these cohorts who entered their respective Academies in kindergarten (in the 2010-11 school year), but left the school since then, that might certainly influence the results (as I mention briefly in the post). That's one of the problems -- one of several -- with using cross-sectional, absolute performance measures to gauge school effectiveness. Thanks for the question, MD

What are your thoughts on Specifically, "Longitudinal analyses have found extremely high rates of attrition within student cohorts and students with disabilities and English Language Learners are over-represented among the students who disappear from Success Academy rosters."

Researchers really need to look at the educational levels of parents who enroll their children in charters like Success Academy and KIPP. California provides that demographic data on parents and the parents of KIPP students often have higher educational levels than the parents of kids in neighborhood schools –and there is plenty of evidence that parent educational level is as strong a predictor of student achievement as family income. I looked but was unable to find demographic data on parents of students in NY. Do you know where to obtain that info?


This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the Shanker Blog may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.