Post post-publication review
A current meme in science is to use the ability with electronic journals and the like for everybody and their cat to weigh in on a scientific paper; this goes by the rather haughty name of post-publication peer review. A prime example is PubPeer, which so far seems to have gained little traction in the solid earth science community. In the wider internet, this often leads to flame wars and other incendiary communications, but there seems to be a hope that this can be an improvement on peer review for scientific publications. In some cases, there is no pre-publication review. But what happens after post-publication review?
Does this really work?
OK, first there is the issue with human nature. Publish a paper that is publicly available and then have somebody point out a flaw, your first instinct might be to defend the paper rather than look to see if you screwed up. As there is no barrier to publication if you don’t reexamine the point in question, it is often a lot faster to just react “no, I got it right, you must be wrong” (GG has seen this even in a review context where there really was an error). And there is usually no need to respond at all if you don’t want to. And then there are some interesting ethical problems that have emerged.
But then there is the issue of just what happens to the science in the paper. Say the authors agree after some months that there was an error. Does this mean the paper is retracted? Amended? If this is serious enough, it is a retraction, but that is pretty rare in earth science. If it is trivial enough (missing labels in a figure, stuff like that) it is often a correction. But maybe the publisher doesn’t amend the published version; do you have to find all the post-publication review spaces to learn of this issue? Perhaps, even, publishers think that this takes them off the hook from bothering to correct such mistakes even if requested by the authors. Maybe articles should have version numbers.
How about if the discussion reveals some useful information that maybe doesn’t have to be in the paper but is enlightening, and say you want to cite that information. What do you do? And is the journal archiving any post-publication peer review they host on their site? Is this the equivalent of a personal communication?
Do we now have to rely on clever Google searches to find all the relevant review on a paper?
Is it possible to plagiarize post-publication review comments? Who is minding that store?
Look, there is always muttering in the peanut gallery about papers that folks care about. When things get serious enough, the usual mode of communication is a comment (usually accompanied by a reply), or publishing a paper directly addressing the concerns on the first paper. The nice thing about these is that they usually have been more carefully thought through than ramblings on a webpage, and they are citable.
In this space GG occasionally engages in what some would call post-publication peer-review, largely to simply take the time to really try and work through a paper that bears on GG’s research interests. Are these comments fair? Not always, and sometimes we’ve seen a few comments pointing out where GG is wrong or differences of opinion. All fine. But should people be relying on these near-random comments on papers as a real substitute for pre-publication peer review? Exactly what is the problem we are trying to solve? Is there bad science that would be exposed by post-publication review that wouldn’t be if there wasn’t such review?