A recent piece from The New York Times “Grey Matter” column in the Sunday Review continues what seems to be a trend toward suggesting science doesn’t work or is somehow malfunctioning.
The author of this latest piece is political scientist Michael Suk-Young Chwe, and the headline, Scientific Pride and Prejudice.
Here’s the lede:
Science is in crisis, just when we need it most.
It’s a good hook, but is it true?
To back up this alarming claim, we get this link to a Nature paper. The paper says that in cancer drug development, too many researchers are jumping ahead into clinical studies before the pre-clinical studies are replicated. And the Nature authors found that of a sample of pre-clinical studies, most could not be reproduced.
Does this mean science is in crisis or that people in certain areas of medical research are not using the best scientific methods? The Nature authors argue for the latter. And whether you think we need better cancer research now than we did in the past probably depends on whether someone you care for has cancer now, or you’ve already lost someone to cancer in some previous year.
Chwe, author of the book Jane Austin, Game Theorist, offers a solution to this crisis/problem. He suggests scientists look to the humanities.
The premise is an intriguing one. After all, scientists can be misled by their all-too-human foibles. And perhaps literary geniuses have insights into those foibles that scientists would find useful. Beyond that, science, like art, requires creativity, and so scientists might find lessons and inspiration from artists and scholars in art.
And yet, in this essay, the only insight we get is that humans are biased. Here’s the view of literary critic E.D. Hirsch.
A claim is not likely to be disproved by an experiment that takes that claim as its starting point. Mr. Hirsch warns about falling “victim to the self-confirmability of interpretations.”
It’s a danger the humanities have long been aware of. In his 1960 book “Truth and Method,” the influential German philosopher Hans-Georg Gadamer argues that an interpreter of a text must first question “the validity — of the fore-meanings dwelling within him.” However, “this kind of sensitivity involves neither ‘neutrality’ with respect to content nor the extinction of one’s self.” Rather, “the important thing is to be aware of one’s own bias.” To deal with the problem of selective use of data, the scientific community must become self-aware and realize that it has a problem.
This is somewhat ambiguous. Is Hersch saying that scientists can’t test their own hypotheses, and therefore science is doomed (suggesting it’s been in crisis since the start) or is Hersch’s statement just a less eloquent way of putting what physicist Richard Feynman said about the number one rule being not to fool yourself?
It’s odd that the piece discusses bias yet fails to discuss the already known importance of independent replication, falsifiability and blinding.
The concept of blinding in scientific investigation goes back at least to Benjamin Franklin (possibly the inventor if it). Along with Antoine Lavoisier, Franklin used blinding to debunk the claims about “animal magnetism” invented by pseudoscientist Franz Mesmer (whose fame lives on in the word mesmerize). In some accounts an actual blindfold was employed. The idea was to test claims by followers of Mesmer that they could detect an alleged magnetic fluid in trees around Franklin’s French residence.
In a similar case, magician James Randi used blinding to debunk a homeopathy-related claim that was published in Nature in the 1980s. Immunologist Jacques Benveniste claimed his experiments showed that water retained the memory of dissolved substances after the substances were diluted to zero concentration. As part of an investigative team, Randi came up with a different blinding method that involved taping to the ceiling an envelope with a key to which flasks held the alleged homeopathic memory. Under the new blinding measure, Benveniste could not replicate his own results.
Note that all parties may have been biased in these cases. Ben Franklin and James Randi may have come in already believing that mesmerism and homeopathy were bunk. But they get the last word and for good reason. The experiments, done properly, gave the answers.
The Pride and Prejudice essay does bring up a couple of examples, the first being the now infamous 2010 claim by Felisa Wolfe-Simon that she’d found a microbe capable of incorporating arsenic in place of the phosphorus in the backbone of its DNA.
But that was a different, deeper kind of failure than Benveniste’s paper. Biologists from the start voiced objections that the experiments were not adequate to demonstrate the claim. The conclusions did not follow from the data. Benveniste at least designed an experiment that when properly blinded could – and did - falsify his claim. Felisa Wolfe-Simon never even accomplished that.
The Times piece wraps up with a mention of bias in Robert Millikan’s famous oil drop experiment, which established the charge of the electron some 100 years ago:
But it would be wrong to say that the ideal scholar is somehow unbiased or dispassionate. In my freshman physics class at Caltech, David Goodstein, who later became vice provost of the university, showed us Robert Millikan’s lab notebooks for his famed 1909 oil drop experiment with Harvey Fletcher, which first established the electric charge of the electron.
The notebooks showed many fits and starts and many “results” that were obviously wrong, but as they progressed, the results got cleaner…in other words, Millikan excluded the data that seemed erroneous and included data that he liked, embracing his own confirmation bias.
This opens up a very important question. Why did the biased Millikan get essentially the right answer (to within 1% of the present value) and end up with a Nobel Prize, while the biased Felisa Wolfe-Simon ended up with a debunked, irreproducible claim? The humanities people have no answer in the Times piece, but physicist Goodstein offers a plausible answer in this fascinating essay I discovered in American Scientist.
In Goodstein’s view, Millikan discounted data points not because they led to an answer he didn’t like, but because he didn’t make the oil drops the right size or otherwise screwed up the technique. This was 100 years ago, after all, and he couldn’t just go out and buy an electron charge measurer. He had to invent the first one and work out the bugs. But it was, according to Goodstein, a good experiment and one that was capable of falsifying the hypothesis that there was a particle carrying a discrete, measurable, minimum unit of electric charge. Felisa Wolfe-Simon, from what biologists say, did not do the right experiments to measure how much, if any arsenic was incorporated in the DNA.
So in the end the scientific Pride and Prejudice essay launches from an intriguing premise but needs something more specific to make the case. There’s nothing wrong with suggesting that scientists read Jane Austen, but at this point it may not be a good substitute for the suggestions of the authors of the Nature paper cited at the beginning. They recommend that drug researchers make sure they use proper blinding, employ appropriate controls, and get independent confirmation of pre-clinical results before moving on to clinical trials.