Performance Measurement In Healthcare And Education
A recent story in the New York Times reports that, according to an Obama Administration-commissioned panel, the measures being used to evaluate the performance of healthcare providers are unfairly penalizing those that serve larger proportions of disadvantaged patients (thanks to Mike Petrilli for sending me the article). For example, if you’re grading hospitals based on simple, unadjusted re-admittance rates, it might appear as if hospitals serving high poverty populations are doing worse -- even if the quality of their service is excellent -- since readmissions are more likely for patients who can’t afford medication, or aren’t able to take off from work, or don’t have home support systems.
The panel recommended adjusting the performance measures, which, for instance, are used for Medicare reimbursement, using variables such as patient income and education, as this would provide a more fair accountability system – one that does not penalize healthcare institutions and their personnel for factors that are out of their control.
There are of course very strong, very obvious parallels here to education accountability policy, in which schools are judged in part based on raw proficiency rates that make no attempt to account for differences in the populations of students in different schools. The comparison also reveals an important feature of formal accountability systems in other policy fields.
In both the education and health care contexts, among the more common objections to adjusting performance measures is the idea that doing so -- e.g., controlling for student/patient characteristics, such as income -- represents “setting different expectations” for institutions (schools/hospitals) based on the people they serve. The Times article, for instance, notes that the Obama Administration commissioned the above-mentioned study, but was “not entirely comfortable with [its] recommendations," since they believe that “performance scores should generally not be adjusted or corrected to reflect differences in the income, race or socioeconomic status of patients."
On the one hand, this is a very well-intentioned point of view. It is indeed unfair, and even bad policy in any accountability system, to hold institutions to different standards, particularly when those standards vary by characteristics of those served. And it is true that adjusting a measure such as re-admittance rates for patients’ income or education could in practice assess the performance of two hospitals as being the same even if one had a higher raw rate than the other.
On the other hand, this viewpoint completely ignores a very important distinction - that between healthcare provider performance and patient health outcomes (this is very similar to the distinction between school and student performance, which we have discussed many times). As is the case with student performance on tests, patients’ health outcomes vary widely, but not all of that variation can be attributed to differences in the performance of health care providers. Many factors that are largely outside of providers’ control, such as patients’ behavior and circumstances, also affect outcomes.
So, in this kind of formal accountability system, we should not be setting expectations for patients' health outcomes per se. Rather, we should be setting expectations for institutions' contributions to those outcomes (i.e., their measured performance).
And those contributions must, to the degree possible, be isolated by the performance indicators used in an accountability system. Measures such as raw re-admittance rates, which reflect patient health outcomes more than they reflect healthcare providers' contributions to those outcomes, don't do that.
Granted, variables such as patients’ income and education are highly imperfect proxies for capturing the differences among patients, and one cannot "control for everything," but simply ignoring these realities by using raw, unadjusted outcomes inevitably penalizes the institutions and providers who serve the disadvantaged populations who most need their services (this is one big reason why education is moving toward more sophisticated growth-based models, though these estimates, like all measures, are imperfect).
In this sense, turning a blind eye to the fact that health outcomes vary due to factors that are outside of providers’ control also represents setting different expectations for performance based on the characteristics of patients being served. And it does so in a manner that disadvantages institutions serving people who need the most help.
I do not feel qualified to offer recommendations for the health care measures – I could come up with a few but they would be rather off-the-cuff, as my background in this area is lacking. The important point here, in addition to the obvious fact that these issues can be found in many non-education policy areas, is that “expectations” and “performance” are not distinct concepts in formal accountability policy. Rather, measuring the performance of complex institutions such as schools and hospitals is exceedingly difficult precisely because it requires that we adjust expectations in accord with circumstances.
Unless we are willing to take drastic and likely unfeasible steps, such as randomly assigning individuals to different hospitals or schools, we must address the fact that raw outcomes capture factors that are outside of the control of those being held accountable, and to do so in turn requires the use of adjustment techniques that must often rely on individual characteristics, such as income and education.
This is not “setting different expectations” in the sense of tolerating low performance, but rather making sure that the measures to which we apply our expectations reflect, to the degree possible, “true performance." What’s more, I’m afraid that our failure to face these facts could act as a powerful disincentive for our most important institutions to serve those who need their services the most, as well as impede their ability to do so effectively.
- Matt Di Carlo