The media are constantly looking for significant new discoveries to feed to the general public who want to know what controls our lives and how to better them. Cut down on salt, eat more vegetables, avoid social networking sites and so on, factors that have been reported to have significant effects on the health and welfare of individuals.
Scientists also seek significance—though it is a technical term that has a different meaning to its common usage. In science, when a discovery is highly significant, it is one that is more likely to reflect a real state of Nature rather than a random fluctuation. However, when society hears that something is significant, this is interpreted as an important finding that has major impact. The problem is that patterns can be highly significant but not very meaningful to the individual. This is where effect size comes into play—a concept that ought to be more widely known.
Calculating effect size (e.g. “Cohen's d”) involves mathematics beyond the scope of this piece but suffice to say, it considers population distributions when estimating how strong an effect is. As such, effect size, rather than significance, is a more meaningful measure of just how important a pattern really is in relation to all the other patterns that influence our lives.
Effect size is best calculated from a number of independent studies to avoid the problems inherent in limited observations. The more scientists studying a phenomenon, the better, as there is less opportunity for errors or mendacious manipulation. A meta-analysis is a study of all the studies that have sought to measure a phenomenon and is the best way to calculate effect sizes. As there are so many factors that can influence your observations—population differences, sampling errors, methodological differences and so on, meta-analysis makes sense to gather together as much evidence to estimate the effect size for a phenomenon.
Consider the reputed difference between the mathematical ability of boys and girls—a highly contentious debate that has even showcased in the pages of this publication. Meta-analyses of over 3 million children have found significant differences between boys and girls in elementary school but that the effect sizes are so small (d less than 0.05) as to be meaningless. The male advantage that does emerge later in schooling is due to factors other than gender that increasingly play a role in mathematical ability.
Humans are complex biological systems affected by a plethora of mechanisms from genetic inheritance to environmental changes that interact in ways that vary from one individual to another and are too complex to map or predict. In an effort to isolate mechanisms, scientists often strip away extraneous variables to reveal the core factors under investigation, but in doing so, create a false impression of the true influence of the mechanism in relation to all the others that play a role. What they find may be significant, but effect size tells you whether it is meaningful.