George Johnson has a new column in The New York Times. It’s called Raw Data, and last week, it offered a big conversation starter – why so many scientific results are not reproduced. Early on, he drops this bombshell:
It has been jarring to learn in recent years that a reproducible result may actually be the rarest of birds.
What follows is an argument very similar to one made in a recent story in The Economist, which ran behind the cover teaser, How Science Goes Wrong, and an older story in the Ne w Yorker by Jonah Lehrer headlined, Is there Something Wrong with the Scientific Method? I wrote somewhat critically on both pieces in this Tracker post last fall.
Johnson’s version of this story includes the contention that most of the good stuff in science has already been done:
With the most accessible truths already discovered, what remains are often subtle effects, some so delicate that they can be conjured up only under ideal circumstances, using highly specialized techniques.
It’s not clear what he means by “truths” and the vagueness of the term made the argument hard to follow. The statement reminded me of some of the arguments John Horgan presented in his 1997 book The End of Science. That book was a fun read, but it did not make a convincing case that science was about to end, and as you may have noticed, science did not end. Science has a way of delivering surprises. Soon after the book was released, people discovered that the expansion of the universe was accelerating. That opened up a whole new slew of scientific questions. And it was reproduced.
There’s no physics mentioned in Johnson’s column, nor is there any astronomy, chemistry or evolutionary biology. Johnson interviews the same primary source featured in The Economist and New Yorker stories – John Ioannidis. He’s described in the Times as a “meta scientist who researches research.” But Ioannides’ profile says he’s an MD and professor of medicine. Does he do meta research in all areas of science, or just medicine? And if he does take aim at non-medical science, is he qualified to do so? These are important questions but the column doesn’t address them. There are no examples outside of medical research.
Ioannidis is quoted indirectly saying today’s scientists are taking long-shots:
Given the desire for ambitious scientists to break from the pack with a striking new finding, Dr. Ioannidis reasoned, many hypotheses already start with a high chance of being wrong.
Is that a problem, or do some areas of science require a lot of trial and error? Is this something new or a description of the way some fields have always been? These questions are not answered.
This is not at all the way physicists describe their situation. They say that they can’t just make up any old hypothesis because they are tightly constrained by quantum mechanics and general relativity. And they’re constrained by umpteen measurements of the way atoms and particles and light behave in the real world. So they can’t get away with just dreaming up long-shot hypotheses without violating some aspect of reality as it’s been measured.
Another consideration is the fact that human bodies are all different, while all electrons, protons and Higgs Bosons are the same, so naturally you won’t get the same kind of “truth” asking about the mass of the electron as you get asking about the risks and benefits of a daily aspirin. The aspirin was one of two specific examples cited in the column, the other being the risks of hormone replacement therapy. But in those cases there may not be one answer that fits all people. That’s not new, and it doesn’t mean medical research in inherently unscientific – or can’t be scientific. It just means consumers need to be skeptical.
The column says the “hottest” fields are more prone to irreproducible results, but does not tell us which fields those are and why they are considered “hot”. And then the column offers the brain twisting contention that Ioannidis uses science to show that most published science is wrong:
Putting all of this together, Dr. Ioannidis devised a mathematical model supporting the conclusion that most published findings are probably incorrect.
To avoid sucking readers into a logic vortex here, this needed a couple of additional pieces of information. First, we don’t know whether he’s talking about just medical literature, or he means to include everything, from climatology journals to the Physical Review. The other missing piece of information is whether scientists consider results published in journals to be “true” or merely interesting enough to merit attention and possible attempts at replication. It’s not really clear what the problem is.
Physicists did observe one famously irreproducible result recently – the faster-than-light-neutrino. But that was not incorporated into the body of physics knowledge. The vast majority of physicists didn’t accept it. The people who discovered it did not try to concoct a new theory to explain it. Instead, they rechecked the experiment and found a cable disconnected. When it was fixed, the result vanished.
This is not to say that science writers should stop being ever-vigilant of hype. It’s misleading to pump up individual findings as new “truths” – whatever that’s supposed to mean.
But over the long haul, many results are replicated – the evidence for many of the exoplanets, the Higgs (two independently run detectors revealed the same thing), the accelerating universe, the existence of antimatter, Bose-Einstein condensate, and the ripples in the cosmic microwave background. Biologists are unlikely to retract the double helix. Chemists are unlikely to declare that benzene molecules aren’t really rings, or that methane isn’t really composed of a carbon atom and four hydrogens. It seems unlikely they will tell us that the sun doesn’t work by nuclear fusion or to discover that fission doesn’t work.
Scientists in many fields build on one another’s findings and so flawed or misinterpreted experiments can get weeded out when other groups try to incorporate them into something new.
The column gives the suggestion that scientist “fool themselves.” I wasn’t sure if that was a reference to a quote by Richard Feynman:
“The first principle is that you must not fool yourself and you are the easiest person to fool.”
I would be very surprised of Ioannidis could find anything wrong with the experimental backing for quantum electrodynamics (For which Feynman won a Nobel Prize).
In the end, it’s not surprising that a vain, status-hungry species of primate would run into limits when attempting to comprehend a sometimes chaotic and pointless-looking universe. What’s surprising, even astonishing, is that we’ve made so much progress and that we’re still going strong.
Leave a Reply