When objections are overruled

A short piece in Ars Technica passes on the inference that scientific rebuttal papers have little impact. The work has some flaws (presuming, for instance, that citation rates would fall when the rebuttal was published, when often the rebuttal is voicing an opinion already established within a skeptical part of the community, or that citations of a rebuttal are all that meaningful) and the recommendation is rather weak tea (there should be links to the rebuttal in the original paper, which is fine as far as it goes and is in fact policy for many online publishers–though the links are not necessarily very prominent).  While some people like to write comments on papers, GG has resisted this urge for many years despite (as should be obvious from many other posts) being deeply skeptical about some work.  Why not write comments?

There are a number of issues writing a comment pointing out weaknesses in published work.  One is that such pieces are inherently negative, which is a bit of a turnoff.  You are asking people to simply forget or unlearn something rather than offering something new.  Another is that usually the original authors will rebut the comment; basically, if there was something appealing about the original paper, the reply will often paper over flaws well enough to allow readers to continue to embrace the thesis advanced in the original paper.  Third, the flaws being pointed out are generally well below the kinds of misbehavior that would justify retraction. Fourth, the venue is a limiting one, where a fully expansive reconsideration of the topic is simply not possible. Fifth, it is very hard not to have a rather hostile tone in a comment (and sometimes the tone is downright offensive), which again can turn readers against the comment rather than the original paper.

(This is, by the ways, part of why GG advocates for solid and thorough review of papers.  In the review stage, one can convince authors that they made some mistakes that could now be corrected; after publication there is more of a tendency to defend rather than revise work as it can be hard to publicly say “whoops, we need a redo”).

In a way this is all unfortunate because of course a robust discussion of the pros and cons of some research should be enlightening.  But the reality is that it is rarely a help.  Should you want to write a comment, though, consider those weaknesses.  It is possible to offer up a positive spin when commenting on some papers, and there is no point in writing anything mean-spirited no matter how many pictures of the lead author remain taped on dartboards in your lab. Offering insights to replace the ones you feel are mistaken would probably be wise.

GG advocates advancing the understanding of the topic at hand by conducting research relevant to the points in dispute.  You can then, armed with new data or analysis, more fully dismember a truly misdirected piece of research, and you are then in the catbird seat should your work inspire a comment. This is obviously a slower sort of response and so might not be suitable in certain circumstances (e.g., a paper influencing important policy decisions), but overall it is more satisfying.  Instead of just destroying somebody’s attempt to add a brick to the edifice of scientific knowledge, you are more in the position of replacing that rotten brick with a sounder one and maybe putting more on top of that. And it has the advantage of keeping you from making a knee-jerk stupid response.  Occasionally you learn that you were in fact wrong (GG is aware of a few instances where a scientist was convinced a line of research was misdirected and set out to prove it wrong only to discover it was right and, by trying to do solid work on it, provided stronger evidence for that line of inquiry).

So maybe it isn’t too tragic that rebuttals seem to be ineffective.  It might just be that the most influential attacks on new science are not appearing in that particular venue.

Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: