Skip to:

In Research, What Does A "Significant Effect" Mean?

Comments

A look at what makes the effect size meaningful (as opposed to significant) one may check here: http://www2.ed.gov/rschstat/eval/choice/implementation/achievementanalysis-sizes.pdf and here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.126.8383&rep=rep1&type=pdf And even when some studies do everything right, and do find significant effects, whether that makes a big difference in reality is another question altogether. Check this one. http://ies.ed.gov/ncee/pubs/20114001/pdf/20114002.pdf

What's your view on Professor Hattie's table 'effect sizes'? Clearly he has gone further than claiming interventions are more then merely 'significant effects' and has tried to quantify these effects. I can accept that there are limits to the type of meta-analyses he conducts, but does that mean we can't trust his research at all? How else should we make decisions on what's effective and what's not?

Great post. I think another thing to consider, unfortunately, is that interpretation of effects often requires some background knowledge of the phenomena being studied. Even other scientists, who can look for effect size, and know that when you have census data, everything is "significant," can still misinterpret things like NAEP scores, where one needs to have an understanding of the history of the test, the content, and the testing situation. But definitely, I am on board with urging caution and humility in interpreting effects.

Thanks for explaining this, I can use this information in conversation and debates. I even have to keep reminding myself not to fall for the rhetoric of people using their ideology to mislead the public with authoritarian, "scientific facts".

And - the size of an effect shouldn't be compared to doing nothing in a schools context as an acton is invariably taken instead of some other possible approach. Educational researchers ought to care about an effect that is greater than the stuff a teacher/school normally does. It's unusual for a set of teachers to do something new and it not to have an effect of some kind. The important question is; is this approach better than the other possible approaches we know about for improving boys reading, for example. Comparison with best known strategy is what good medical research undertakes. There really isn't any point in demonstrating that a particular strategy improves boys reading unless the strategy is better than the best one presently known.

DISCLAIMER

This web site and the information contained herein are provided as a service to those who are interested in the work of the Albert Shanker Institute (ASI). ASI makes no warranties, either express or implied, concerning the information contained on or linked from shankerblog.org. The visitor uses the information provided herein at his/her own risk. ASI, its officers, board members, agents, and employees specifically disclaim any and all liability from damages which may result from the utilization of the information provided herein. The content in the Shanker Blog may not necessarily reflect the views or official policy positions of ASI or any related entity or organization.