What is “Publication”?
Thinking a bit more on post-publication review and what not and it occurs to GG that the term “publication” is in flux. In the past, a publication was a fixed thing. The publication I read when the journal first came out 20 years ago is identical to the one you might read 20 years from now. But if we take the premise of post-publication peer review seriously, just what is a publication?
This might be getting to the heart of what peer review is good for. GG’s impression of post-publication review is that it is nearly always focused on “is this paper correct?” or occasionally, “is it fraud?” (since there is little such formal post-publication review in earth science, this might be a misunderstanding). As such, there is certainly value in this activity. But what action, if any, should the author take upon receiving such review? The extremes–leave the paper as is or retract it–exist in the non-post-publication-review world. But what if an author changes a figure, or adds a new section?
Many of us are already familiar with the need to not only cite a source on the web but to indicate the access date as some such materials change over time. Currently, though, this refers to archived datasets or documentation for software or some such material. Is this where we want the scientific literature to go? Are we to cite “Smith et al., The Great Paper, v2.1”? Might we end up citing different versions of the same paper in a single work? Are all the versions to be archived? Scientists change their minds: Einstein, for instance, viewed the cosmological constant in some of his earlier work as a huge mistake. Would he have edited it out and made it vanish from the literature? Or to take an earth science example, Joseph LeConte was an early advocate for a glacial origin for Yosemite Valley, but later in life he decided he was wrong and Yosemite was a graben. Would it have benefited the field for him to have removed or replaced the set of observations he made earlier? Or in these cases, was it better to have both their early thoughts and their later ones available for consideration?
If post-publication review doesn’t allow for changes to a paper–just a sort of global thumbs up or down–then how is it really peer review? It does not provide the author an opportunity to refine the paper to better present its points, or to clarify text that was murky. It is basically a social media version of the old-school chat over beer at a meeting leading folks to decide to ignore a flawed paper, or the writing of a formal comment. Evaluating the merits of a published paper is not a new thing. Are we gaining enough by having this kind of review as we lose the permanence of “publication”?
And even if post-publication review journals allow for changes, how common are reviews that seek to improve the communication the author sought? It seems like a lot of the discussions GG has seen focus on disputes, not on clarifying the paper. Is this a useful replacement for pre-publication, traditional peer review? After all, post-publication reviewers are self-selected and unburdened by the need to consider the paper as a whole.
Some journals are exploring an open review arrangement, where a paper exists as a pre-publication open to all while still soliciting formal reviews. This arrangement has several merits: somebody will definitely look at the paper, and if it is of broad interest, lots more feedback will be accumulated (perhaps allowing the author to better refute misunderstandings of the formal reviewers). The author presumably is expected to respond to feedback so obtained, but what if the decision of the editor is rejection? Was the pre-publication a publication? Was it being cited even as it was reviewed? Does this arrangement obviate the desire for post-publication review? For controversial work, will such papers be covered by the popular media before they are fully vetted? Could we end up generating more anti-vaccine type nonsense by having such an arrangement?
Peer-review and scientific publication is at a crossroads, and there are profound challenges across the board. But the full implications of all the solutions need to be considered. We need to identify what we need the scientific literature to look like and then work back to how it can be made to be so. Adding thumbs-up and thumbs-down buttons and a comments section isn’t necessarily a positive development….