Limits of Peer Review

Many times peer review is touted in the popular press as something like the Good Housekeeping Seal of Approval; if it was peer-reviewed, it must be right.  The misperception is based on a misunderstanding of what reviewers are really doing, but it is increasingly evident that even the classical purpose of peer review is getting buried by increasingly complex and arcane procedures that a review cannot really examine.

First, the perception issue.  Peer review is really not deciding if the conclusions in a paper are right or wrong.  GG will point out flaws in an interpretation, but so long as the paper has presented new data or new analyses, the paper merits publication even if the conclusions seem overblown or misguided. So what exactly is a peer review doing?  Well, you want to see that there is new material, that the way it was collected and analyzed is presented clearly, and that the analysis was appropriate and correctly completed. So ideally a peer-reviewed paper is one where, if they say “we determine summer paleotemperatures from this proxy were 20 +/- 5°,” there is a pretty good chance that will stand up, but the interpretation of such a value in terms of, say, elevation might be up for grabs.  A paper does not have to present all the possible alternative interpretations (though if the authors want their interpretation believed, they will address such alternatives).

Sometimes peer review is portrayed as an adversarial process.  Although this does happen from time to time (after all, this is a human endeavor), most authors welcome a careful vetting of their work.  Far better to learn of an algebraic mistake prior to publication than after publication.  Also, authors are far more open to altering their interpretation prior to publication; once you have said in print something, it can be harder to decide that wasn’t right.

Now as peer review is mainly about what was actually done for a scientific project, how is that itself being threatened? Well, consider a seismic topography paper saying “We took 10,000 seismograms, picked body wave arrival times using standard  cross-correlation software NNN and analyzed them with software ABC. The resulting images are in figures X,Y, and Z.” Frequently (as in nearly always) there is no supplemental file with the seismograms (that might eat several gigabytes), nor one with the picked times, and all too frequently none with the resulting velocity structure. It is likely that there are a number of parameters that should be set to use software NNN and ABC of which we know nothing. In this case, we don’t know the version of the software, and academic software all too frequently has multiple origins and multiple versions at any one time. How can a reviewer know if things were done right? After all, there is no way to redo the calculations, and heaven help you if software packages are proprietary or incompatible with computers you have access to.  At most a reviewer might notice oddities in the images provided that might suggest some blunder somewhere, but by and large the whole process has become an enormous black box. So at this point the poor reviewer really can say nothing more than “sounds like this was done right” or “seems to be missing a step.” Basically the more we see this kind of paper, the less useful peer review will be. All too often, in the absence of essential information, a reviewer might fall back on their perception of what should be the result which can lead to reviews approving bad work that supports an initial bias or rejecting good work that conflicts with other ideas.

There are partial solutions.  Ask for data to be presented in tables; seek out innovative ways of displaying raw data.  Explain the connection between data and image (or conclusion) by showing the data or some subset in some way where the connection to the final image is clear.