When sharing with me the results of some tests, my doctor once said, "You are a scientist, you know a single piece of data can't provide all the answers or suffice to make a diagnosis. We can't look at a single number in isolation, we need to look at all results in combination." Was my doctor suggesting that I ignore that piece of information we had? No. Was my doctor deemphasizing the result? No. He simply said that we needed additional evidence to make informed decisions. This is, of course, correct.
In education, however, it is frequently implied or even stated directly that the bottom line when it comes to school performance is student test scores, whereas any other outcomes, such as cooperation between staff or a supportive learning environment, are ultimately "soft" and, at best, of secondary importance. This test-based, individual-focused position is viewed as serious, rigorous, and data driven. Deviation from it -- e.g., equal emphasis on additional, systemic aspects of schools and the people in them -- is sometimes derided as an evidence-free mindset. Now, granted, few people are “purely” in one camp or the other. Most probably see themselves as pragmatists, and, as such, somewhere in between: Test scores are probably not all that matters, but since the rest seems so difficult to measure, we might as well focus on "hard data" and hope for the best.
Why this narrow focus on individual measures such as student test scores or teacher quality? I am sure there are many reasons but one is probably lack of familiarity with the growing research showing that we must go beyond the individual teacher and student and examine the social-organizational aspects of schools, which are associated (most likely causally) with student achievement. In other words, all the factors skeptics and pragmatists might think are a distraction and/or a luxury, are actually relevant for the one thing we all care about: Student achievement. Moreover, increasing focus on these factors might actually help us understand what’s really important: Not simply whether testing results went up or down, but why or why not.
For the past ten months we've been highlighting this body of research in “the social side of education” series. This has been an uphill battle for a variety of reasons: 1) the research in this area seems less likely to be picked up by the popular media; 2) the research is in a variety of different disciplines and employs different approaches within disciplines, which requires a lot of synthesizing work; and 3) in order to pay attention to this evidence (let alone be persuaded by it), a mind shift is necessary. We are so used to thinking about individual, rational actors who respond to incentives and disincentives that it's just hard to imagine anything else.
But there is some indication -- or at least I am hopeful -- that our collective ability to view school performance and improvement more broadly, and in a more complex way, is expanding. For one, many researchers doing this work are eager to share it, and concerned about its policy impact. Second, while I may once again be exhibiting motivated optimism, I do detect some nascent attention to these issues from journalists and commentators. For example, EdWeek's Stephen Sawchuk recently highlighted the work of scholars John P. Papay and Matthew A. Kraft, whose analysis showed that "teachers who work in more supportive environments become more effective at raising student achievement on standardized tests over time than do teachers who work in less supportive environments."
A recent New York Times piece by Motoko Rich included the following quote from NYU professor Pedro Noguera:
If you only look at the numbers, and you don’t probe and look at the learning environment, the culture of the school or the relationships between teachers and students, you’re going to miss out on a lot.
Furthermore, I find more and more recognition not only that context and relationships matter in schools, but also that they can be quantified, and even used in accountability systems. In other words, it's not necessarily numbers versus all the other "squishy stuff." Researchers have made immense progress in devising complex, creative ways to measure these constructs -- see here for an overview. And, again, the result of this work is unequivocal: What happens at the organizational and interpersonal level (e.g., relationships, collaboration, trust) shapes key outcomes like teacher effectiveness and student achievement.
New York City's new school report card is another reason for optimism. In addition to student achievement, the City's new "School Quality Snapshot" attempts to provide information about the broader school community and about what happens in this community at the interpersonal level (e.g., school-family ties, trust, leadership and collegiality etc.). A simple graphic summarizing the basic elements of this system is below.
Some commentators, however, have argued that focusing on these aspects deemphasizes attention to what really matters: Student testing results. In a recent post, the Fordham Institute’s Robert Pondiscio argued:
Other elements of the draft Snapshot are equally warm and fuzzy. Sure, “rigorous instruction” is on top, but “student achievement,” like the proverbial pony, is buried beneath a pile of collaborative teachers, supportive environments, and strong family-community ties. In short, this document feels driven more by philosophy than data, relying on qualitative measures of uncertain value while strongly de-emphasizing student achievement in general and student growth specifically.
The claim that non-test measures, such as those of trust and relationships, are "philosophy," while test-based measures are "data," is, put simply, incorrect. Are student test scores and data from self-reported surveys the same kind of measure? No. Are they both examples of quantitative data? Yes (measured on the same four category scale, in fact). Are they both proxies for constructs that are far more complex? Yes. Are they both valuable? Yes.
(For the record, the report cards in fact do not seem to use qualitative measures, though they might consider doing so, as these too can be very useful.)
As mentioned earlier, social-relational aspects of schools can be (and increasingly are) measured and analyzed quantitatively in a variety of ways. Of course doing this well isn't easy, and much of what we'd be able to learn from the measures would depend on how the variables and indicators are treated. And then of course there is the issue of how various pieces of data are weighted in decision making, which I do not believe the NYC report cards even does (thus, there is no emphasizing or deemphasizing here).
So, yes, measuring context and relationships within schools is not straightforward, but neither is measuring student learning. The difference is we are more familiar and comfortable with the notion that test scores are a proxy for student learning than with the idea that responses to a survey can capture trust. But more comfortable does not mean more right.
We have a choice. We can continue to measure only what we are familiar with -- e.g., student learning using standardized tests -- or we can broaden what we measure -- e.g., non-cognitive student outcomes, social aspects of schools etc. All are valuable, and should play a role in school improvement. It is a huge mistake to portray school environmental and contextual variables as somehow non-scientific or “squishy,” especially since we have the tools to develop nuanced and tailored school measures and corresponding improvement strategies.
Finally, I also think we need to know more about how data are shared back with stakeholders. For example, recent research tells us that different kinds of feedback have different effect on teachers’ ability to improve -- see Taylor and Tyler (2012) and Blazar and Kraft (forthcoming) -- which strongly suggests that sharing data for improvement needs to be done in certain ways and not others.
In fact, we know that attending to and understanding the social lay of the land (and here) is essential for the implementation of any reform. In sum, as the aforementioned NY Times article notes, we have a lot of data. My questions are: 1) do we have the most important data; and 2) do we know how to use them? My answers: 1) probably not, but we could gather them if we wanted to, and that's happening in some places; and 2) Yes, methods to understand these data are rapidly improving. However, we need to learn more about how these data are best shared and used with stakeholders for school improvement.