Post-publication peer review in public: poison or progress?
What a difference a few days make. The much-ballyhooed story of arsenic-utilizing bacteria stemming from NASA’s embargoed press conference and paper in Science (somewhat ballyhooed here by yours truly), is accumulating criticism from microbiologists calling into question the degree of rigor applied to some of the paper’s experiments.
[Update: For an excellent, unexplored take on the analytical chemistry used in the paper, see this upcoming C&EN article by my colleague here, Dr. Carmen Drahl.]
On Sunday, I added to my post the first chink in the armor: a detailed technical critique by University of British Columbia microbial geneticist, Dr. Rosemary (Rosie) Redfield. Dr. Redfield has long found value in blogs and has encouraged her trainees to have their own blogs to openly discuss their science.
I’ve done some of the techniques (awhile ago) that Redfield discusses from the paper but it even took me some time to go through her critique. The central theme of her criticisms is that the experimental results leading to the conclusion that the Mono Lake GFAJ-1 bacterium can grow using arsenic instead of phosphorus may be an artifactual: the detection of arsenic in the bacterial DNA could have been due to insufficient clean-up of the DNA prior to ICP-MS analysis and that trace amounts of phosphate in the media and from dead cells could have provided the remaining cells with enough phosphate to survive in 40 mM arsenate. For example, the former artifact might result because the authors used phenol-chloroform to extract bacterial DNA from agarose gel slices which might have carried over some non-specifically bound arsenic that might not have occurred had a Gene-Clean type glass beads purification been done instead.
Here’s an excerpt of her conclusions:
Bottom line: Lots of flim-flam, but very little reliable information. The mass spec measurements may be very well done (I lack expertise here), but their value is severely compromised by the poor quality of the inputs. If this data was presented by a PhD student at their committee meeting, I’d send them back to the bench to do more cleanup and controls.
There’s a difference between controls done to genuinely test your hypothesis and those done when you just want to show that your hypothesis is true. The authors have done some of the latter, but not the former.
A CBC News article on Monday spoke further on Redfield’s critique.
As painful as her analysis might have been to read for the authors, NASA, and editors of Science, Redfield’s critique represents the future of post-publication peer-review. Already the present at journals such as those from PLoS where readers can comment on papers at the journal website, open public discussion of science immediately upon release of a paper will help science progress more quickly. No longer confined to the archaic correspondence to the journal that takes months to view, the internet and the science blogosphere is facilitating open discussion of peer-reviewed publications within hours and days of release.
And this is where I follow-up on my longstanding position that all scientists should have a blog of some sort that, at the very least, can capture the content and discussion of journal clubs. Of course, the frank discussion at journal clubs requires that one also be willing to open up one’s own work to similarly rigorous criticism.
So, it’s still somewhat unusual for the legacy scientific publishing establishment to encounter such criticism from a senior scientist such as Redfield. But it was the response of NASA and the authors that has provoked much head-shaking among the scientific community.
From the CBC News article cited above:
When NASA spokesman Dwayne Brown was asked about public criticisms of the paper in the blogosphere, he noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that Wolfe-Simon will not be responding to individual criticisms, as the agency doesn’t feel it is appropriate to debate the science using the media and bloggers. Instead, it believes that should be done in scientific publications.
It’s difficult to get more hypocritical than this – the story was hyped for days and promoted by press release and press conference. Yes, the paper was subjected to peer review by one of the world’s leading journals but – but dear god, people – grow a thicker skin if you expect to hump a research finding publicly but not expect that it will be subjected to rigorous post-review analysis by scientists who know how to use the internet and set up a Blogger or WordPress blog.
This is a call to pre-Enlightenment thinking. Brown is telling us to judge utterances not by their content, not even by the integrity, reputation, and experience of the individuals who deliver them, but by whether they’re delivered from the proper place in the proper building — in pre-Enlightenment days, the Church of Rome; in Brown’s post-arsenic days, the Church of the Peer-reviewed Journal.
It’s an extraordinary dismissal. Rosie Redfield is a full-bore member of the academy and a researcher in the field under question. She is — to extend the metaphor — a priest. But though Redfield wears the proper robes, Brown wants to dismiss her because she’s not standing on the proper altar.
Earlier this morning via Twitter, Ivan Oransky of Embargo Watch pointed out that comments of this sort accumulating at his blog post sound similar to the editorial of Royce Murray’s we covered a few weeks ago.
Yesterday, Carl Zimmer wrote in Slate, “This Paper Should Not Have Been Published,” after interviewing nearly a dozen experts in the field – but not two of the paper’s authors who instead provided comment by e-mail.
“We cannot indiscriminately wade into a media forum for debate at this time,” declared senior author Ronald Oremland of the U.S. Geological Survey. “If we are wrong, then other scientists should be motivated to reproduce our findings. If we are right (and I am strongly convinced that we are) our competitors will agree and help to advance our understanding of this phenomenon. I am eager for them to do so.”
“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated,” wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. “The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”
Zimmer is not at the altar – he’s just one of the most prominent and decorated science writers of our time. In fact, Oransky called him, “the dean of my generation of science writers.” But don’t you see, Carl? Asking for comment “do[es] not represent the proper way to engage in a scientific discourse.”
Again, from Zimmer’s Slate article:
While Redfield considers Wolfe-Simon’s research “flim-flam,” she thinks it’s fine for the NASA scientists to hold off responding to their critics. She is working on a formal letter to Science detailing her objections. But Jonathan Eisen of UC-Davis doesn’t let the scientists off so easily. “If they say they will not address the responses except in journals, that is absurd,” he said. “They carried out science by press release and press conference. Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.”
Let it be then. The former water cooler, conference room, and pub critiques of scientific work are now taking place in the widely-accessible and, yes, messy venue of online communication. If authors don’t wish to engage and defer discussion for a few weeks and let letters to be sent to Science and published, that’s their prerogative. Perhaps Dr. Redfield should chisel her letter on a stone tablet. I’m sure that a telegraph line might still run from Vancouver to Washington, DC.
In defense of Science, Oransky opened his post yesterday by quoting a much more reasoned and thoughtful comment from AAAS director of public programs Ginger Pinholster who responded to his post on December 3rd as to how the story took on a life of its own.
I don’t envy the authors – my observation has been that NASA put them in a difficult situation by how they hyped the December 2 press conference as revealing, “an astrobiology finding that will impact the search for evidence of extraterrestrial life.”
Me? I can assure you of one thing: We’re going to go back and triple-check some of our own data before sending in our next revised manuscript.
And no press conference.
UPDATE: Literally minutes after I scheduled this post to go up, Dr. Felissa Wolfe-Simon put up this post at her professional website:
Wolfe-Simon et al Comment: 08 December 2010
My research team and I are aware that our peer-reviewed Science article has generated some technical questions and challenges from within the scientific community. Questions raised so far have been consistent with the range of issues outlined by journalist Elizabeth Pennisi in her Science news article, which was published along with our research. For instance, other scientists have asked whether the bacteria had truly incorporated arsenic into their DNA, and whether the microbes had completely stopped consuming phosphorus. Our manuscript was thoroughly reviewed and accepted for publication by Science; we presented our data and results and drew our conclusions based on what we showed. But we welcome lively debate since we recognize that scholarly discourse moves science forward. We’ve been concerned that some conclusions have been drawn based on claims not made in our paper. In response, it’s our understanding that Science is in the process of making our article freely available to the public for the next two weeks to ensure that all researchers have full access to the findings. We invite others to read the paper and submit any responses to Science for review so that we can officially respond. Meanwhile, we are preparing a list of “frequently asked questions” to help promote general understanding of our work.
Although much of the attention has been on Rosie Redfield’s initial critique, Alex Bradley’s guest post at We, Beasties on ScienceBlogs appeared the next day and added to the discussion of the hydrolytic lability of arsenic esters in putative As-containing DNA.