In August 2015, Brian Nosek and the Open Science Collaboration published a report on the replicability of findings previously published in top rank psychology journals:
“We conducted replications of 100 experimental and correlational studies... using high-powered designs and original materials when available.”
Only 36 percent of the replications were "successful." Among the findings that didn't replicate were these:
“People are more likely to cheat after they read a passage informing them that their actions are determined and thus that they don't have free will.”
“People make less severe moral judgments when they've just washed their hands.”
“Partnered women are more attracted to single men when they're ovulating.”
These particular findings may not be game-changing. But they have been widely cited by other researchers (including me).
In many cases there may well be innocent explanations for why the original study gave the unreliable results it did. But in more than a few cases it can only be put down to slipshod research, too great haste to publish, or outright fraud. Worryingly, the more newsworthy the original finding, the more likely it could not be replicated. Insiders have likened the situation to a “train-wreck.”
John Brockman likes to quote Stewart Brand:
“Science is the only news. When you scan through a newspaper or magazine, all the human interest stuff is the same old he-said-she-said, ... a pathetic illusion of newness. Human nature doesn't change much; science does.”
But we have here a timely reminder that the distinction between science and journalism is not—and has never been—so clear-cut as Brand imagines.
The reality is that science itself has always been affected by “this human interest stuff.” Personal vendettas, political and religious biases, stubborn adherence to pet ideas have in the past led even some of the greatest scientists to massage experimental data and skew theoretical interpretations. Happily, the body of scientific knowledge has continued to live and grow despite such human aberrations. In general scientists continue to play by the rules.
But we must not be complacent. The professional culture is changing. In many fields, and not of course only in psychology, science is becoming more of a career path than a noble vocation, more of a feeding trough than a chapel of truth. Sub-prime journals are flourishing. Bonuses are growing. After the disgrace of the bankers, science must not be next.