Confirmation Bias and the Science of Psychology

Researchers famously tried to replicate 100 prominent psyhology studies and reported last year in Science that only about 40 percent could be. This scandalous result has recently provoked a defensive backlash from other practitioners who asserted that the Science replicability study itself was flawed and that the reported results of psychological research are, in fact, stronger than its critics have claimed. Even as the controversy has been playing out, a new study finds that one of the more robust results of psychological research may be bunkum. Specifically, the new study to be published in Perspectives on Psychological Science challenges the finding that an individual’s willpower declines as he or she resists various immediate temptations.

During the past two decades, the ego depletion effect has been confirmed by scores of psychological studies using all manner of experimental techniques. However, according to Slate, a new study using 2,000 subjects in experiments across the world that tried to replicate ego depletion has found absolutely no such effect. This is a huge bombshell. As Slate explains:

No sign that the human will works as it’s been described, or that these hundreds of studies amount to very much at all. …

For scientists and science journalists, this back and forth [over replicability] is worrying. We’d like to think that a published study has more than even odds of being true. The new study of ego depletion has much higher stakes: Instead of warning us that any single piece of research might be unreliable, the new paper casts a shadow on a fully-formed research literature. Or, to put it another way: It takes aim not at the single paper but at the Big Idea.

[Roy] Baumeister’s theory of willpower, and his clever means of testing it, have been borne out again and again in empirical studies. The effect has been recreated in hundreds of different ways, and the underlying concept has been verified via meta-analysis. It’s not some crazy new idea, wobbling on a pile of flimsy data; it’s a sturdy edifice of knowledge, built over many years from solid bricks.

And yet, it now appears that ego depletion could be completely bogus, that its foundation might be made of rotted-out materials. That means an entire field of study—and significant portions of certain scientists’ careers—could be resting on a false premise. If something this well-established could fall apart, then what’s next? That’s not just worrying. It’s terrifying.

The whole Slate article is well worth your attention.

In his seminal 2005 article, “Why Most Published Research Findings Are False,” Stanford University statistician John Ioannides concluded that “for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.”

Might I suggest that psychology and, in fact, the rest of science would benefit from some really focused research on the problem of confirmation bias?

For more background, see my Reason feature “Broken Science.“