One might think the tale has been told and done to death on the famous, speculative paper asserting that microbes in Mono Lake, CA, have adapted to their low-phosphorus and high-arsenic environment by swapping the latter atoms into their DNA and other molecules of life for the former ones. But at USA Today Dan Vergano this week published a story that adds a genuinely illuminating end note. It pertains to a great deal more than this one well-meant and earnestly presented (if almost certainly wrong in primary thrust) paper and its publication two-plus years ago in the illustrious magazine Science.
What his investigation does is to zero in on the opinions the journal's editors received from the outside experts they consulted. Vergano does not report how he got his hands on them. Such reviews are generally kept anonymous and out of the public eye, although exceptions are rife. Lead authors themselves get them, of course, and from there co-authors, grad students and post-docs and deans. Maybe even the lunch buddies of all those people may be handed them too. The peer review system's security measures are hardly CIA-grade.
*Correction: Dan Vergano got the info via FOIA request to NASA. It's in the story, I missed it through three readings. Erase all my speculating, just above. Thanks also to commenters who cordially noted the error. Had I gotten this aspect correct the first time through, I'd have speculated on why exactly NASA – even though it ran a press conference – had info on the contents of the peer review. But I'm quitting while I'm not farther behind than already.
More important, it somewhat gets Science's internal judges off the hook for deeming the paper not only sufficiently novel (that, for sure) but so convincing (not, as it happens, so much) that it not only made their cut but wound up starring in a full-bore NASA press conference. That is, somebody should have said this paper needs to be published, but not by us, and maybe they should have told the author it should just be a speculative poster paper somewhere for a lot more peers to take a crack at.
The reviews, Vergano reports, were highly positive, even ecstatic. He has a few selected passages from them as evidence. It takes discipline to dog this angle so hard, given that it's a mop-up story after the main news has come and gone. Kudos. The story's importance is not so much to explain how this paper got in but how porous and erratic is the peer review system more broadly. Short of paying high ranking experts to spend a very long time examing and deconstructing highly complex papers submitted for publication (which would divert a lot of smart people from doing useful research of their own), one has little to offer as remedy. It is a voluntary system. Its unpaid main laborers must fit the reviewing into often-busy and stressful lives. It actually works pretty well. Medical journals are particularly in a tough spot, given the big pharma financial factors and the conscious and unconscious biases they create.
Vergano's story is fairly tough on Science, quoting people who think its peer review system managers should have upped the bar considerably for a paper so audacious in its conclusions. But one must say that this one instance does not mean Science's peer review is no better than the process at other, lesser journals. Better doesn't mean perfect, especially with something as slippery and subjective as ruling on somebody else's work and often in a hurry. This one slipped through. Science perhaps has already circulated a lessons-learned memo. But more incidents like this surely will follow anyway. And to see bits from the peer reviewers' reports is a sneaky, second-hand pleasure.
Caveat lector.
Leave a Reply