Do Subgroup Accountability Measures Affect School Ratings Systems?

The school accountability provisions of No Child Left Behind (NCLB) institutionalized a focus on the (test-based) performance of student subgroups, such as English language learners, racial and ethnic groups, and students eligible for free- and reduced-price lunch (FRL). The idea was to shine a spotlight on achievement gaps in the U.S., and to hold schools accountable for serving all students.

This was a laudable goal, and disaggregating data by student subgroups is a wise policy, as there is much to learn from such comparisons. Unfortunately, however, NCLB also institutionalized the poor measurement of school performance, and so-called subgroup accountability was not immune. The problem, which we’ve discussed here many times, is that test-based accountability systems in the U.S. tend to interpret how highly students score as a measure of school performance, when it is largely a function of factors out of schools' control, such as student background. In other words, schools (or subgroups of those students) may exhibit higher average scores or proficiency rates simply because their students entered the schools at higher levels, regardless of how effective the school may be in raising scores. Although NCLB’s successor, the Every Student Succeeds Act (ESSA), perpetuates many of these misinterpretations, it still represents some limited progress, as it encourages greater reliance on growth-based measures, which look at how quickly students progress while they attend a school, rather than how highly they score in any given year (see here for more on this).

Yet this evolution, slow though it may be, presents a somewhat unique challenge for the inclusion of subgroup-based measures in formal school accountability systems. That is, if we stipulate that growth model estimates are the best available test-based way to measure school (rather than student) performance, how should accountability systems apply these models to traditionally lower scoring student subgroups?

Social And Emotional Skills In School: Pivoting From Accountability To Development

Our guest authors today are David Blazar and Matthew A. Kraft. Blazar is a Lecturer on Education and Postdoctoral Research Fellow at Harvard Graduate School of Education and Kraft is an Assistant Professor of Education and Economics at Brown University.

With the passage of the Every Student Succeeds Act (ESSA) in December 2015, Congress required that states select a nonacademic indicator with which to assess students’ success in school and, in turn, hold schools accountable. We believe that broadening what it means to be a successful student and school is good policy. Students learn and grow in multifaceted ways, only some of which are captured by standardized achievement tests. Measures such as students’ effort, initiative, and behavior also are key indicators for their long-term success (see here). Thus, by gathering data on students’ progress on a range of measures, both academic and what we refer to as “social and emotional” development, teachers and school leaders may be better equipped to help students improve in these areas.

In the months following the passage of ESSA, questions about use of social and emotional skills in accountability systems have dominated the debate. What measures should districts use? Is it appropriate to use these measures in high-stakes setting if they are susceptible to potential biases and can be easily coached or manipulated? Many others have written about this important topic before us (see, for example, here, here, here, and here). Like some of them, we agree that including measures of students’ social and emotional development in accountability systems, even with very small associated weights, could serve as a strong signal that schools and educators should value and attend to developing these skills in the classroom. We also recognize concerns about the use of measures that really were developed for research purposes rather than large-scale high-stakes testing with repeated administrations.

Building A Professional Network Of Rural Educators From Scratch

Our guest author today is Danette Parsley, Chief Program Officer at Education Northwest, where she leads initiatives like the Northwest Rural Innovation and Student Engagement Network. To learn more about this work, check out Designing Rural School Improvement Networks: Aspirations and Actualities and Generating Opportunity and Prosperity: The Promise of Rural Education Collaboratives.

Small rural schools draw from a deep well of assets to positively impact student experiences and outcomes. They tend to serve as central hubs within their communities, and their small size often facilitates close staff relationships, which in turn can enable moving innovative ideas into action. At the same time, rural schools face a number of challenges that differ from those of their urban and suburban counterparts.

First, it’s extremely difficult to draw high-quality teachers to geographically disconnected, rural communities—and, when they do come, it’s hard to get them to stay. Second, it’s a challenge to connect teachers across remote and rural communities so they can share instructional practices and professional development. One way to address the challenges facing rural schools, while leveraging their inherent assets, is to establish professional networks of teacher leaders aimed at providing support that helps their colleagues succeed and encourages them to stay.

The Details Matter In Teacher Evaluations

Throughout the process of reforming teacher evaluation systems over the past 5-10 years, perhaps the most contentious, discussed issue was the importance, or weights, assigned to different components. Specifically, there was a great deal of debate about the proper weight to assign to test-based teacher productivity measures, such estimates from value-added and other growth models.

Some commentators, particularly those more enthusiastic about test-based accountability, argued that the new teacher evaluations somehow were not meaningful unless value-added or growth model estimates constituted a substantial proportion of teachers’ final evaluation ratings. Skeptics of test-based accountability, on the other hand, tended toward a rather different viewpoint – that test-based teacher performance measures should play little or no role in the new evaluation systems. Moreover, virtually all of the discussion of these systems’ results, once they were finally implemented, focused on the distribution of final ratings, particularly the proportions of teachers rated “ineffective.”

A recent working paper by Matthew Steinberg and Matthew Kraft directly addresses and informs this debate. Their very straightforward analysis shows just how consequential these weighting decisions, as well as choices of where to set the cutpoints for final rating categories (e.g., how many points does a teacher need to be given an “effective” versus “ineffective” rating), are for the distribution of final ratings.

An Alternative Income Measure Using Administrative Education Data

The relationship between family background and educational outcomes is well documented and the topic, rightfully, of endless debate and discussion. A students’ background is most often measured in terms of family income (even though it is actually the factors associated with income, such as health, early childhood education, etc., that are the direct causal agents).

Most education analyses rely on a single income/poverty indicator – i.e., whether or not students are eligible for federally-subsidized lunch (free/reduced-price lunch, or FRL). For instance, income-based achievement gaps are calculated by comparing test scores between students who are eligible for FRL and those who are not, while multivariate models almost always use FRL eligibility as a control variable. Similarly, schools and districts with relatively high FRL eligibility rates are characterized as “high poverty.” The primary advantages of FRL status are that it is simple and collected by virtually every school district in the nation (collecting actual income would not be feasible). Yet it is also a notoriously crude and noisy indicator. In addition to the fact that FRL eligibility is often called “poverty” even though the cutoff is by design 85 percent higher than the federal poverty line, FRL rates, like proficiency rates, mask a great deal of heterogeneity. Families of two students who are FRL eligible can have quite different incomes, as could two families of students who are not eligible. As a result, FRL-based estimates such as achievement gaps might differ quite a bit from those calculated using actual family income from surveys.

A new working paper by Michigan researchers Katherine Michelmore and Susan Dynarski presents a very clever means of obtaining a more accurate income/poverty proxy using the same administrative data that states and districts have been collecting for years.

A Small But Meaningful Change In Florida's School Grades System

Beginning in the late 1990s, Florida became one of the first states to assign performance ratings to public schools. The purpose of these ratings, which are in the form of A-F grades, is to communicate to the public “how schools are performing relative to state standards.” For elementary and middle schools, the grades are based entirely on standardized testing results.

We have written extensively here about Florida’s school grading system (see here for just one example), and have used it to illustrate features that can be found in most other states’ school ratings. The primary issue is the heavy reliance that states place on how highly students score on tests, which tells you more about the students the schools serve than about how well they serve those students – i.e., it conflates school and student performance. Put simply, some schools exhibit lower absolute testing performance levels than do other schools, largely because their students enter performing at lower levels. As a result, schools in poorer neighborhoods tend to receive lower grades, even though many of these schools are very successful in helping their students make fast progress during their few short years of attendance.

Although virtually every states’ school rating system has this same basic structure to varying degrees, Florida’s system warrants special attention, as it was one of the first in the nation and has been widely touted and copied (as well as researched -- see our policy brief for a review of this evidence). It is also noteworthy because it contains a couple of interesting features, one of which exacerbates the aforementioned conflation of student and school performance in a largely unnoticed manner. But, this feature, discussed below, has just been changed by the Florida Department of Education (FLDOE). This correction merits discussion, as it may be a sign of improvement in how policymakers think about these systems.

A Myth Grows In The Garden State

New Jersey Governor Chris Christie’s recently announced a new "fairness funding" plan to provide every school district in his state roughly the same amount of per-pupil state funding. This would represent a huge change from the current system, in which more state funds are allocated to the districts that serve a larger proportion of economically disadvantaged students. Thus, the Christie proposal would result in an increase in state funding for middle class and affluent districts, and a substantial decrease in money for poorer districts. According to the Governor, the change would reduce the property tax burden on many districts by replacing some of their revenue with state money.

This is a very bad idea. For one thing, NJ state funding of education is already about 7-8 percent lower than it was in 2008 (Leachman et al. 2015). And this plan would, most likely, cut revenue in the state’s poorest districts by dramatic amounts, absent an implausible increase in property tax rates. It is perfectly reasonable to have a discussion about how education money is spent and allocated, and/or about tax structure. But it is difficult to grasp how serious people could actually conceive of this particular idea. And it’s actually a perfect example of how dangerous it is when huge complicated bodies of empirical evidence are boiled down to talking points (and this happens on all “sides” of the education debate).

Pu simply, Governor Christie believes that “money doesn’t matter” in education. He and his advisors have been told that how much you spend on schools has little real impact on results. This is also a talking point that, in many respects, coincides with an ideological framework of skepticism toward government and government spending, which Christie shares.

The Intervention That Works Across Settings With All Children

Our guest authors today are Geoff Marietta, Executive Director, Pine Mountain Settlement School and Research Fellow at Berea College; Chad d'Entremont, Executive Director, Rennie Center for Education Research & Policy; and Emily E. Murphy, Director, Massachusetts Education Partnership (MEP) at the Rennie Center. Their work focuses on research and practice in labor-management-community collaboration.

If you learned there was an intervention to improve student outcomes that worked for nearly all children across communities, what would stop you from using it? This intervention has closed learning gaps, both in urban communities serving predominantly low-income minority students and in isolated rural areas with large numbers of white and Native American students living in poverty. It has worked in suburban, urban, and rural settings with white, African-American, Hispanic, Native American, Asian, and multi-racial students. That intervention is collaboration.

In this post, we define collaboration, briefly discuss the growing evidence associating collaboration with student success, and describe some of our ongoing work, which focuses on designing tools to facilitate, formalize, and focus the hard but worthwhile and necessary responsibility of working together.

Teachers' Opinions Of Teacher Evaluation Systems

The primary test of the new teacher evaluation systems implemented throughout the nation over the past 5-10 years is whether they improve teacher and ultimately student performance. Although the kinds of policy evaluations that will address these critical questions are just beginning to surface (e.g., Dee and Wyckoff 2015), among the most important early indicators of how well the new systems are working is their credibility among educators. Put simply, if teachers and administrators don’t believe in the systems, they are unlikely to respond productively to them.

A new report from the Institute of Education Sciences (IES) provides a useful little snapshot of teachers’ opinions of their evaluation systems using a nationally representative survey. It is important to bear in mind that the data are from the 2011-12 Schools and Staffing Survey (SASS) and the 2012-13 Teacher Follow Up Survey, a time in which most of the new evaluations in force today were either still on the drawing board, or in their first year or two of implementation. But the results reported by IES might still serve as a useful baseline going forward.

The primary outcome in this particular analysis is a survey item querying whether teachers were “satisfied” with their evaluation process. And almost four in five respondents either strongly or somewhat agreed that they were satisfied with their evaluation. Of course, satisfaction with an evaluation system does not necessarily signal anything about its potential to improve or capture teacher performance, but it certainly tells us something about teachers’ overall views of how they are evaluated.

New Research Report: Are U.S. Schools Inefficient?

At one point or another we’ve all heard some version of the following talking points: 1) “Spending on U.S. education has doubled or triped over the past few decades, but performance has remained basically flat; or 2) “The U.S. spends more on education than virtually any other nation and yet still gets worse results.” If you pay attention, you will hear one or both of these statements frequently, coming from everyone from corporate CEOs to presidential candidates.

The purpose of both of these statements is to argue that U.S. education is inefficient - that is, gets very little bang for the buck – and that spending more money will not help.

Now, granted, these sorts of pseudo-empirical talking points almost always omit important nuances yet, in some cases, they can still provide important information. But, putting aside the actual relative efficiency of U.S. schools, these particular statements about U.S. education spending and performance are so rife with oversimplification that they fail to provide much if any useful insight into U.S. educational efficiency or policy that affects it. Our new report, written by Rutgers University Professor Bruce D. Baker and Rutgers Ph.D. student Mark Weber, explains why and how this is the case. Baker and Weber’s approach is first to discuss why the typical presentations of spending and outcome data, particularly those comparing nations, are wholly unsuitable for the purpose of evaluating U.S. educational efficiency vis-à-vis that of other nations. They then go on to present a more refined analysis of the data by adjusting for student characteristics, inputs such as class size, and other factors. Their conclusions will most likely be unsatisfying for all “sides” of the education debate.

Subscribe to Education Policy