Sample Size And Volatility In School Accountability Systems

It is generally well-known that sample size has an important effect on measurement and, therefore, incentives in test-based school accountability systems.

Within a given class or school, for example, there may be students who are sick on testing day, or get distracted by a noisy peer, or just have a bad day. Larger samples attenuate the degree to which unusual results among individual students (or classes) can influence results overall. In addition, schools draw their students from a population (e.g., a neighborhood). Even if the characteristics of the neighborhood from which the students come stay relatively stable, the pool of students entering the school (or tested sample) can vary substantially from one year to the next, particularly when that pool is small.

Classes and schools tend to be quite small, and test scores vary far more between- than within-student (i.e., over time). As a result, testing results often exhibit a great deal of nonpersistent variation (Kane and Staiger 2002). In other words, much of the differences in test scores between schools, and over time, is fleeting, and this problem is particularly pronounced in smaller schools. One very simple, though not original, way to illustrate this relationship is to compare the results for smaller and larger schools.

The Debate And Evidence On The Impact Of NCLB

There is currently a flurry of debate focused on the question of whether “NCLB worked.” This question, which surfaces regularly in the education field, is particularly salient in recent weeks, as Congress holds hearings on reauthorizing the law.

Any time there is a spell of “did NCLB work?” activity, one can hear and read numerous attempts to use simple NAEP changes in order to assess its impact. Individuals and organizations, including both supporters and detractors of the law, attempt to make their cases by presenting trends in scores, parsing subgroups estimates, and so on. These efforts, though typically well-intentioned, do not, of course, tell us much of anything about the law’s impact. One can use simple, unadjusted NAEP changes to prove or disprove any policy argument. And the reason is that they are not valid evidence of an intervention's effects. There’s more to policy analysis than subtraction.

But it’s not just the inappropriate use of evidence that makes these “did NCLB work?” debates frustrating and, often, unproductive. It is also the fact that NCLB really cannot be judged in simple, binary terms. It is a complex, national policy with considerable inter-state variation in design/implementation and various types of effects, intended and unintended. This is not a situation that lends itself to clear cut yes/no answers to the “did it work?” question.

The Increasing Academic Ability Of New York Teachers

For many years now, a common talking point in education circles has been that U.S. public school teachers are disproportionately drawn from the “bottom third” of college graduates, and that we have to “attract better candidates” in order to improve the distribution of teacher quality. We discussed the basis for this “bottom third” claim in this post, and I will not repeat the points here, except to summarize that “bottom third” teachers (based on SAT/ACT scores) were indeed somewhat overrepresented nationally, although the magnitudes of such differences vary by cohort and other characteristics.

A very recent article in the journal Educational Researcher addresses this issue head-on (a full working version of the article is available here). It is written by Hamilton Lankford, Susanna Loeb, Andrew McEachin, Luke Miller and James Wyckoff. The authors analyze SAT scores of New York State teachers over a 25 year period (between 1985 and 2009). Their main finding is that these SAT scores, after a long term decline, improved between 2000 and 2009 among all certified teachers, with the increases being especially large among incoming (new) teachers, and among teachers in high-poverty schools. For example, the proportion of incoming New York teachers whose SAT scores were in the top third has increased over 10 percentage points, while the proportion with scores in the bottom third has decreased by a similar amount (these figures define “top third” and “bottom third” in terms of New York State public school students who took the SAT between 1979 and 2008).

This is an important study that bears heavily on the current debate over improving the teacher labor supply, and there are few important points about it worth discussing briefly.

Resources On The Social Side Of Education Reform

Updates to this post will be posted here. 

For the past few months, we have been insisting, through this blog series, on the idea that education reform has a social dimension or level that often is overlooked in mainstream debate and policy. Under this broad theme, we've covered diverse issues ranging from how teachers' social capital can increase their human capital to how personnel churn can undermine reform efforts, or how too much individual talent can impede a team's overall performance. This collection of issues may prompt a number of important questions: What exactly is the "social side?" What are its key ideas? I would like to offer a few initial thoughts and share some resources that I've compiled.

The social side is primarily a lens that brings into focus a critical oversight in the public debate on educational reform and its policies: The idea that teaching and learning are not solo but rather social endeavors that are achieved in the context of the school organization, and within the districts where schools are embedded, through relationships and teamwork, rather than competition and a focus on individual prowess. 

This social side perspective does a few things:

Constitution For Effective School Governance

Our guest author today is Kenneth Frank, professor in Measurement and Quantitative Methods at the Department of Counseling, Educational Psychology and Special Education at Michigan State University.

Maybe it’s because I grew up in Michigan, but when I think of how to improve schools, I think about the “Magic Johnson effect." During his time at Michigan State, Earvin “Magic” Johnson scored an average of 17 points per game. Good, but many others have had higher averages. Yet, I would want Magic Johnson on my team because he made everyone around him better. Similarly, the best teachers may be those that make everyone around them better.  This way of thinking is not currently the focus of many current educational reforms, which draw on individual competition and market metaphors.

So how can we leverage the Magic Johnson effect to make schools better? We have to think of ways that teachers can work together. This might be in terms of co-teaching, sharing materials, or taking the time to engage one another in honest professional dialogues. There is considerable evidence that teachers who can draw on the expertise of colleagues are better able to implement new practices. There is also evidence that when there is an atmosphere of trust teachers can engage in honest dialogues that can improve teaching practices and student achievement (e.g., Bryk and Schneider, 2002).

Feeling Socially Connected Fuels Intrinsic Motivation And Engagement

Our "social side of education reform" series has emphasized that teaching is a cooperative endeavor, and as such is deeply influenced by the quality of a school's social environment -- i.e., trusting relationships, teamwork and cooperation. But what about learning? To what extent are dispositions such as motivation, persistence and engagement mediated by relationships and the social-relational context?

This is, of course, a very complex question, which can't be addressed comprehensively here. But I would like to discuss three papers that provide some important answers. In terms of our "social side" theme, the studies I will highlight suggest that efforts to improve learning should include and leverage social-relational processes, such as how learners perceive (and relate to) -- how they think they fit into -- their social contexts. Finally, this research, particularly the last paper, suggests that translating this knowledge into policy may be less about top down, prescriptive regulations and more about what Stanford psychologist Gregory M. Walton has called "wise interventions" -- i.e., small but precise strategies that target recursive processes (more below).

The first paper, by Lucas P. Butler and Gregory M. Walton (2013), describes the results of two experiments testing whether the perceived collaborative nature of an activity that was done individually would cause greater enjoyment of and persistence on that activity among preschoolers.

Librarians, Libraries, Serendipity And Passion

Our guest author today is Connie Williams, a National Board Certified Teacher librarian at Petaluma High School in Petaluma, CA, past president of the California School Library Association, and co-developer of the librarian and teacher 2.0 classroom tutorials.

Down the road from where I live, on the first-of-the month, a group of vintage car owners gather for a “cars and coffee” meet up. The cars that show up with their drivers cover many years and obsessions. Drivers park, open up the car hoods and take a few steps back and begin talking with other car owners and visitors who happen by. These are people who are interested in the way cars work, their history, and they all have stories to share.

How do they know so much about their cars? They work on them – gaining insight by hands-on practice and consultations with experts. If they’re wealthy enough, they pay someone else to do the work, yet they don’t just hand over their cars to them. They read about them, participate in on-line groups, ask for guidance, and they drive them. Most often, when they drive them, someone stops and asks questions about their cars and they teach what they know to others. 

This is an example of the kind of learning we would hope for, for all our students – a passion that is ignited and turns into knowledge that is grown, developed, and shared. In this sense, it is inquiry – asking questions and taking the required steps to answer them – that is at the heart of learning.

The Equity Projection

A new Mathematica report examines the test-based impact of The Equity Project (TEP), a New York City charter school serving grades 5-8. TEP opened up for the 2009-10 school year, receiving national attention mostly due to one unusual policy: They paid teachers $125,000 per year, regardless of experience and education, in addition to annual bonuses (up to $25,000) for returning teachers. TEP largely makes up for these unusually high salary costs by minimizing the number of administrators and maintaining larger class sizes.

As is typical of Mathematica, the TEP analysis is thorough and well-done. The school's students' performance is compared to that of similar peers with a comparable probability of enrolling in TEP, as identified with propensity scores. In general, the study’s results were quite positive. Although there were statistically discernible negative impacts of attendance for TEP’s first cohort of students during their first two years, the cumulative estimated test-based impact was significant, positive and educationally meaningful after three and four years of attendance. As always, the estimated effect was stronger in math than in reading (estimated effect sizes for the former were very large in magnitude). The Mathematica researchers also present analyses on student attrition, which did not appear to bias the estimates substantially, and they also show that their primary results are robust when using alternative specifications (e.g., different matching techniques, score transformations, etc.).

Now we get to the tricky questions about these results: What caused them and what can be learned as a result? That’s the big issue with charter analyses in general (and with research on many other interventions): One can almost never separate the “why” from the “what” with any degree of confidence. And TEP, with its "flagship policy" of high teacher salaries, which might appeal to all "sides" in the education policy debate, provides an interesting example in this respect.

Where Al Shanker Stood: Quick Fixes In Education

In this "Where We Stand" column, which was printed in the New York Times on March 27, 1983, Al Shanker quotes historian Paul Gagnon to argue that we need to think long-term about the purposes of public schooling and agree on a carefully chosen set of education reform priorities. Failing this, they warn, the U.S. will forever be caught in a churn of futile, quick-fix reform initiatives.

It never fails. Whenever there's an educational problem, there's always an attempt to solve it with a quick fix. The current problem - the shortage of science and math teachers - is no exception. A quick fix just won't work. Of course, there are a few things that can be done to ease the problem. The most promising short-run idea is to encourage teachers already teaching in other fields but who have a good background in math and science to switch.

But we won't solve the problem until we know why we have one. It is not just that private industry pays more. It's that there aren't enough students graduating from college in these fields to satisfy the needs of business and the teaching profession. Most students stay away from math and science in college because they didn't get enough of a background in high school. Why? Because math and science course are more difficult than many electives, and most high school students, given a choice between tough courses and easy ones, choose the latter. And it doesn't start there. It goes back to elementary school, and not just with respect to math and science but with the ability to read problems and think them through ... willingness to discipline oneself, to work long and hard.

Building And Sustaining Research-Practice Partnerships

Our guest author today is Bill Penuel, professor of educational psychology and learning sciences at the University of Colorado Boulder. He leads the National Center for Research in Policy and Practice, which investigates how school and district leaders use research in decision-making. Bill is co-Principal Investigator of the Research+Practice Collaboratory (funded by the National Science Foundation) and of a study about research use in research-practice partnerships (supported by the William T. Grant Foundation). This is the second of two posts on research-practice partnerships - read the part one here; both posts are part of The Social Side of Reform Shanker Blog series.

In my first post on research-practice partnerships, I highlighted the need for partnerships and pointed to some potential benefits of long-term collaborations between researchers and practitioners. But how do you know when an arrangement between researchers and practitioners is a research-practice partnership? Where can people go to learn about how to form and sustain research-practice partnerships? Who funds this work?

In this post I answer these questions and point to some resources researchers and practitioners can use to develop and sustain partnerships.

Subscribe to Education Policy