Does anybody benefit from peer review?

Looking at a piece recently about peer review and you get the impression that this is the most worthless exercise imaginable, full of selfish reviewers damning papers for no good reason other than to preserve their own place in the world, failing to fix the multiple problems in manuscripts, and generally just taking up space and time.  Let’s blow the whole thing up!

GG gets the feeling that these articles (and they seem to show up about once a week somewhere; Retraction Watch tends to cite them in their weekly roundup) are things some editors at some websites and publications sit about and try to solicit.  So it seems like people are encouraged to write these by being told: “Have you had problems with peer review? Tell us your horror stories! Blow the lid off this sleazy operation! Get the revenge you’ve been yearning for against all those ungrateful bastards who made you add useless prose and cite irrelevant papers! We’ll share it with the world!”

GG would like to suggest a different question, and one far more appropriate and to the point.  “Have you ever learned of a major failing within one of your manuscripts and been able to correct it because of peer review?” If the answer is truly and honestly no, you can feel free to blare your trumpet about how worthless peer review is.  But if the answer is yes, would you have been happier making that mistake in print (or in publicly available electrons)?  And yes, GG has gotten feedback in review that, in one paper, resulted in some errors in equations being fixed, and in another in a section being added that really did need to be there for the points being advocated to make sense (that from a review recommending rejection, by the ways).  GG thinks he helped two authors avoid publishing papers with actual errors by reviewing material carefully (but you’d really need to ask those authors out of GG’s earshot). Peer review is misunderstood, not only by the public, which mistakes it for some kind of seal of approval, but by lots of scientists, who seem to view it as an opportunity to squash others’ opinions. What it should be is an opportunity to help other scientists present their observations and insights most clearly and effectively so that they can be the basis for new work.

In that piece that inspired this rant, one of the authors basically said that nobody else wrote useful reviews.  Although it was a bit tongue-in-cheek, there was a sense that this was kind of true. GG has had the chance to see a lot of reviews over the years and this is either total BS or there are fields where all the practitioners need a good spanking.  GG has seen inspired, careful, thorough reviews that arguably were worth publication themselves (these gems are perhaps the best argument for some kind of public version of peer review).  Are they rare?  Sure. But they show how positive peer review can be.

Advertisements

Tags: ,

14 responses to “Does anybody benefit from peer review?”

  1. binay panda says :

    no, i don’t think peer review is misunderstood, may be by a v handful. what, instead, is rightly understood how the process of peer-review works at present. of course, reviewing one’s work by peers is always a good idea. we, as scientists, age learning to accept criticism. how can we deviate from the first principle of science? therefore, it will be appropriate to pose a slightly different question. does the peer-review system, as it’s currently practiced, work? and the answer is clearly no.

    the current system of giving away scientific credit is undemocratic, feudal and unscientific, period. http://ow.ly/Qv7Yk. therefore, we need to improve the way the peer-review system works, not abolish it. what we need is a system of post-publication peer-review that is transparent, fair and helps to advance society and not a few individuals’ career or ambitions. thank you. binay panda

    Like

    • cjonescu says :

      Thank you for your thoughts. I did notice you didn’t answer my question in your comment, though from the tone, I’d guess peer review has not helped you. That is too bad.
      By saying that you are “giving away scientific credit” in the current system, do you mean that reviewers who make contributions to improve a paper are being overlooked, or that reviewers are stealing from papers they review? (Yes, I read the linked post and still am not sure who steals from whom).
      Most of the grief with the publication system seems to come from bioscience, where retractions are rife, egos are large, and dishonest behavior is far too common (read Retraction Watch for a bit if you disagree). We in earth science seem to be in a better place at the moment. While peer review has real problems (and I’ve written a little on that before), a wide open system lacking organized review could get to be a real mess. While high-profile results that are of immediate interest get close post-publication attention (regardless of the presence of a formal post-publication system, by the ways), it is the stuff that doesn’t garner such intense attention that would be most hurt by abandoning peer review.
      I’ve long felt that we need to bifurcate the publication system, one branch being exactly what you promote: a free-for-all, citable, no barrier public depository of work, but the other branch being properly peer-reviewed work. The first branch could absorb all the stuff that needed to see the light of day to satisfy grant agencies or bean counters in university administration (not to mention the science not sexy enough for many journals and the failed attempts that should be documented somewhere); the second would return to a place where more thoughtful and complete pieces of work intended to last more than a year or so might reside.

      Like

      • binay panda says :

        and i repeat, i am not against peer review. rather, what i am advocating is a completely different system of peer review. you are absolutely right, and that is, the system is not filled by grief, but broken completely in the field of biology. let me give you an example: most folks in biology spend a considerable amount of time in changing the format of the manuscript (completely nonsensical and non-scientific work in changing the format, references, sticking to the word limit, writing the same cover letter with different names etc. etc.) when they keep sending the same manuscript to many journals after getting each round of rejection. now let me answer your question directly. have i got benefited from the peer-review? of course yes but that’s not the right question to ask. the right question is should i, and others in my group, have benefited more from our manuscripts being reviewed by multiple non-selective group of scientists in an open and transparent manner? and the answer is a resounding yes. you can ask the same question to any randomly picked scientist (i mean genome/computational biology researcher) and he/she wont give you a much different answer. what we need is to broaden the scope of the peer review process that is in practice today. why do you want your manuscript to be reviewed by only 2 or 3 pre-selected scientists? in the internet age, we got to be using technology to aid us, and not stick to an ancient way to judge science. would you agree? may be this is more relevant to our country, india. we have ≥500million kids below the age of 25yrs (yes, thats unbelievable but true). how, i, as a scientist, tell each of those kids who are interested in science that science is a level-playing field, when it isn’t, and attract the best talent to have fun rather than being a part of the rat race. i want science to be a fun profession and not a bone-crushing, highly competitive, heart-rate increasing race to get more grants, tenure, publish in nature & science type of profession. i would love to get your take. thank you. binay panda

        Like

      • cjonescu says :

        I sense some flaws in your logic, binay panda. You presume review “by multiple non-selective group of scientists in an open and transparent manner,” but isn’t it possible to write a paper that *nobody* will “peer review”? Why do you think these are non-selective, anyways? While the selection might not be by an editor, there is selection going on as not everybody who might review a paper will. And how thorough are post-publication peer reviews? Do they help to point out flaws in the presentation? Do they suggest figures that might be useful?
        I am sorry to hear that you are frequently having to reformat papers multiple times to get them published. I do not know if that is a characteristic of your field as a whole or if that is poor judgement on your part in choosing journals to submit to. If journals in your field are trying to show the lowest possible time from submission to publication, then it is probably the field (journals that pride themselves on quick turnaround and little else will get that quick turnaround by only accepting as submitted or rejecting, which is stupid). I will say my experience is that, other than Science and Nature, which are always crap shoots, usually I am able to get published in the journal I initially submit to.
        BTW, you will find in my older posts advocacy for signed peer review; some of what you seem to complain of appears to be poor behavior by anonymous reviewers. Some other thoughts on post-publication review are at https://grumpygeophysicist.wordpress.com/2015/04/26/post-post-publication-review/

        Like

  2. binay panda says :

    i guess we agree on most points. the real diff comes from our individual experience of the peer review system. i am v happy, even surprised, to know that you have had great experience with it and have had a great deal of success in publishing your manuscripts in your first attempt, after peer review, with the journals where you send the manuscripts to. this, perhaps, is the nature of the field of research and i can only be jealous of your field of research. most, from our field will bitterly complain and tell a different version of the same story that i am telling. many a times i wonder, if i am doing science or working for a journal editor in reformatting the same manuscript!!!

    i have read your earlier post. thank you for providing the link. i think we are arguing on a topic where there is no silver bullet. the post publication peer review system is not perfect and honestly was never given a chance to evolve. imho, an evolved, graded, transparent and open post-publication review system is the only solution. how, who and when are the questions that all of us have to figure out together. thank you. binay panda

    Like

    • Andrew says :

      These things are likely to be field dependent. Honestly, as an astronomer, when I read complaints from other fields they’re nothing like what I experience. Manuscripts and references are formatted by LaTeX/BibTeX, and are none of my concern. I’ve never had to submit a paper to a second journal to get it published (well, I once sent an email to Nature asking if they were interested in my results, and they said no). I’ve never written a cover letter for a paper. Nor have a really encountered an unreasonable reviewer (one I disagreed with, sure, but genuinely unreasonable, I don’t think so).

      Signed peer review *might* help with the occasional awful review, but it sounds a lot more like a way to ensure anyone with any power never has to face a meaningful review again. Post-publication review sounds like an excellent way to ensure everyone but the top few groups/institutions get ignored. Double-blind review has some merit (even if often it’s completely obvious who the authors are). But open? I’d refuse to ever do a review for someone who might hold a grudge (i.e., everyone I’m not friends with!) They’re going to be reviewing my grant applications and paper – if they’re petty enough to be a jerk during a review, I’ll bet they’re petty enough to hold a grudge come time to rate my proposal.

      Like

  3. Israel Nelken says :

    My overall experience with peer review is really positive. Of course papers got rejected, sometimes painfully. But reviews also made many of my papers much better. Here are two examples:
    1. Bitterman et al. 2008 (Nature) – a first version (that was rejected) contained an analysis regarding differences between neural responses to speech and music which reviewers pointed out could not be supported by the data we had. They were right, and the section was removed.
    2. Yaron et al. 2012 (Neuron) – the review came back saying more or less ‘you showed differences going one direction, but you would have submitted the paper too if the differences would have gone the other direction – can you explain better what is going on?’ So we went back to run additional experiments, ending with a much stronger result (last couple of figures of the paper).

    Like

  4. SB says :

    Whilst I can’t say I have had a transformational peer review on one of my papers, I did once review a paper that was really very poor – the underlying flaw in the paper was that they had tried to make the data fit their hypothesis, which it just didn’t. In fact the data pointed to a far more interesting theory which I pointed out. The paper was rejected by the journal but I saw it published in a higher impact journal about a year later with some additional data and some of my comments from my review used almost verbatim in the discussion. Hey ho.

    Like

  5. Andrew says :

    I had a reviewer (actually, two) force me to re-write my code almost completely so it could do some standard validations, rather than the non-standard validations I had been using. The science didn’t change, but a) casual readers are now far more likely to be convinced my code works correctly, and b) it can now handle tougher problems that it couldn’t have before, so I’m doing follow-up studies I wouldn’t have been able to do otherwise.

    And, I think, it runs ~100X faster, so it’s a lot easier to do idle simulations to see if something’s worth pursuing.

    Like

  6. CarolynS says :

    I can’t offhand think of an example where peer review has made a big difference to me but it is helpful at the very least to show when something is hard to understand and can be rewritten to be better.

    As myself a frequent peer reviewer, however, I think my reviews have sometimes been pretty helpful at least to the journals to show when some article really has a major problem, undetected or ignored by the authors, and should not be published. These include some kind of misinterpretation (I remember one article that looked great until I finally realized that the significant differences between the various arms was due to changes in the control group not in the intervention group!) and not too infrequently just ignoring the published literature on the same topic from the same data set. I imagine the authors did not feel this was helpful, but I think it did help the journal editors.

    Like

  7. TorbjörnB says :

    They are useful almost every time!

    …although if pressed, I would have to agree that most of the time the flaws pointed out to me have been pretty minor. And also that I have seen some rather unreasonable comments.

    There is however one major exception when I had written up a very technical method paper and expressed what we did in a way that somewhat differed from how people had expressed these things before. All my co-authors, all very knowledgeable people who (yes, also the guy at the end) had all read the manuscript carefully, thought it looked good, and yet it turned out that it was not understandable at all to other experts. Very unusual and I still can’t quite understand how we could so completely miss the target, but peer review certainly was very useful to catch it.

    I also have another story when I had included a section where my new method showed some promising results for a class of problems different from the one my method was designed for. I over-enthusiastically boasted this great success, until a sensible reviewer pointed out that my dataset really was way too small to say anything very general on the topic. Removed section, stayed on topic and published a sharper and better paper.

    Like

    • newcleckitdominie says :

      I think TorbjörnB’s first story illustrates one of the main benefits of peer review: it’s a chance to test-drive your paper and ensure it makes sense to other members of the community. As I work in interdisciplinary areas (I’m a mathematical modeller), it’s very useful to have someone from the other side of the disciplinary boundary giving their perspective on issues such as the terminology I use or the significance I attach to theoretical results.

      Four other good experiences, off the top of my head:

      (i) A reviewer suggested that we try fitting our theoretical predictions to a data set published decades before, which we hadn’t previously heard of. Slightly to our surprise, the prediction fitted the data nicely; this paper is now my most cited by some margin.

      (ii) A reviewer suggested that our work wasn’t suitable for the journal we’d originally submitted to, which is quite prestigious in the maths community but not beyond. S/he suggested an alternative journal with a more geophysical slant, where it was eventually accepted and has almost certainly had more exposure.

      (iii) A reviewer broke the bounds of anonymity to offer us a set of field photos which nicely illustrated a phenomenon we were trying to explain.

      (iv) A reviewer pointed out a hidden assumption in a model, which led to the manuscript being rejected; exploring this assumption we discovered a whole set of issues that hadn’t been commented in in the literature before. (We’re about to submit a new manuscript on this topic.)

      Maybe I’ve just been lucky enough to work in a field where at least some reviewers are quite charitable to the ignorant, but I’m sure that on balance my papers have benefited a lot more than they’ve suffered from peer review!

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: