An Alternative Income Measure Using Administrative Education Data
The relationship between family background and educational outcomes is well documented and the topic, rightfully, of endless debate and discussion. A students’ background is most often measured in terms of family income (even though it is actually the factors associated with income, such as health, early childhood education, etc., that are the direct causal agents).
Most education analyses rely on a single income/poverty indicator – i.e., whether or not students are eligible for federally-subsidized lunch (free/reduced-price lunch, or FRL). For instance, income-based achievement gaps are calculated by comparing test scores between students who are eligible for FRL and those who are not, while multivariate models almost always use FRL eligibility as a control variable. Similarly, schools and districts with relatively high FRL eligibility rates are characterized as “high poverty.” The primary advantages of FRL status are that it is simple and collected by virtually every school district in the nation (collecting actual income would not be feasible). Yet it is also a notoriously crude and noisy indicator. In addition to the fact that FRL eligibility is often called “poverty” even though the cutoff is by design 85 percent higher than the federal poverty line, FRL rates, like proficiency rates, mask a great deal of heterogeneity. Families of two students who are FRL eligible can have quite different incomes, as could two families of students who are not eligible. As a result, FRL-based estimates such as achievement gaps might differ quite a bit from those calculated using actual family income from surveys.
A new working paper by Michigan researchers Katherine Michelmore and Susan Dynarski presents a very clever means of obtaining a more accurate income/poverty proxy using the same administrative data that states and districts have been collecting for years.
The idea is simple yet, to my knowledge, has not been applied previously. Instead of measuring income by whether or not a student is eligible in any given year, Michelmore and Dynarski differentiate between students who are persistently eligible over all years and those whose eligibility is transitory – that is, they are eligible in some years and not others. Using Michigan data, they find that among the universe of eighth grade students who are eligible for FRL in at least one year since they entered the system (which is 60 percent of all students), about one quarter are persistently eligible – i.e., eligible in every year.
Black, Latino, and youth in urban schools were overrepresented among those students who were found to be persistently eligible, as opposed to their more sporadically eligible peers. The former also score more poorly on standardized math tests by 0.27 standard deviations, which is a large discrepancy. And, finally, using this alternative, longitudinal measure, achievement gaps (between the persistently disadvantaged and non-disadvantaged) are 35-40 percent larger than those estimated in the traditional manner. The gap persists even when controlling for prior achievement and other school and student characteristics.
This suggests, put simply, that the students characterized as disadvantaged by the traditional point-in-time FRL eligibility measure are, as expected, a heterogeneous group, and traditional FRL eligibility may be seriously understating the association between income and a wide range of meaningful outcomes.
Although this analysis focuses on a single state, one that has been particularly hard hit in recent years, Michelmore and Dynarski seem to provide an improved alternative to single year FRL eligibility, a measure that is horribly inadequate for its central role in education policy and research. In other words, years of eligibility for FRL may be a pretty good proxy for income, and, at the least, it is better than whether or not students are eligible in any given year.
And this alternative, importantly, can be calculated in any state that can link students between years (which is most states). This means that incorporating the new measure(s) into, for instance, accountability systems would not require much in terms of data infrastructure. It is also a simple calculation, one that is accessible to policymakers, parents, and other stakeholders. As is often the case, experimentation with or adoption of this measure would probably have to start on the state level, and changing institutionalized measurement in policy is never easy.
Still, should future research produce results similar to those reported by Michelmore and Dynarski, persistent disadvantage has the potential to partially rectify a persistent shortcoming in education research and policy.