GG doesn’t do Twitter.  Or Facebook. They just feel like ways to spend time on trivia and allow you to risk defining yourself to the world by electronic immortalization of an inappropriate brain fart. But if Richard Price of Academia.edu has his way, we’d all be doing more social media. He expresses his logic thusly:

A journal editor receives a submission, then asks two academics to review it. The article is then published and read by the larger academic community: depending on the paper, 500 or 1,000 people may read it. What did they think? This 
is lost data that a peer-review system should pick up. Academics, as they read a paper, are noticing aspects of it that they deem robust, and aspects they deem weak. The more of these opinions we can surface, the more we can develop a view of the strength of an individual paper.

OK, so just how are these 500-1000 readers supposed to provide feedback? Swipe left? Click on an upturned thumb? GG isn’t sure, because to sign up with Academia.edu, you have to allow them to access your contacts. Really? [Hmm.  Maybe this is why they have investors instead of grant or foundation support]. If there is any feedback visible to non-members, GG couldn’t see it without joining. How often do you fill out those pop-up surveys from all sorts of websites to “tell us how we are doing”? If you say “never,” odds are, you won’t be providing much feedback here either.

[An aside.  Academia.edu is treading on very shaky ground.  GG has found copyrighted material on this website that he is sure was not approved (uploader had nothing to do with the original authorship or publisher). GG found one paper from 1994 he is a coauthor on in the database quite imperfectly listed (a five author list was shown as “Last author + 1” for author; it wasn’t clear who uploaded this old paper; elsewhere the same paper shows up with the first author listed as the sole author). A paper by Wakabayashi and Sawyer is listed as being authored by Saucedo Rosy. So material isn’t vetted in the most cursory fashion and the boilerplate about not posting copyrighted material isn’t really having any impact. The search engine for this site is really not up for even the most basic challenge. Why would anybody spend any time in this house of mirrors?]

So we’ll proceed on the assumption somebody sometime might at least code the bloody website in a useful manner and pay somebody to limit the copyright violations. In solid earth science, GG isn’t sure that most papers even get scanned by 1000 readers, let alone actually read by that many (the numbers on the Academia.edu site are mostly 0 reads for earth science papers despite listing thousands of followers). And if you have to score a paper you want to read? Probably read less. But what kind of discussions is this invitation likely to produce? Carefully thought out analysis?

Getting a good review as an editor is almost as hard as making one as a reviewer. (By “good” GG means thorough and useful). Mind, this is from professional colleagues identifiable to the editor who have agreed to take on this chore. Good reviews take time–frequently days. Odds are, any commentary on papers in the context of a social network website will be more superficial than even a so-so review. If you are scanning material looking for something in particular for your own work, are you going to stop and read a whole paper and provide feedback? No; maybe you’d say “has useful data”.  Are you going to spend the 10-15 minutes to write down what you thought was strong and what was weak? The Magic 8 Ball says “Don’t count on it”. It isn’t remotely clear to GG that this will yield anything very useful.  Pretending this is a major advance on peer review is kind of delusional.

A lot of these kind of proposals really are assuming that knee-jerk, in-the-moment analysis will greatly improve science, just as short lines of text spread through social media has deepened our political discourse. Although something like this will at times produce the kind of meaningful dialog that improves the science [we will for the moment put aside the problem of citing living documents], more often GG expects that crickets will be the most common sound in that comments area, followed by the occasional “did you notice your latitudes are wrong” kind of comment and then the “how come you didn’t cite my classic paper?” diatribe. Will seat-of-the-pants momentary evaluation of a paper by hundreds uncover the true worth of a paper? Maybe sometimes. GG suspects it will more often reflect the flavor-of-the-month group think in some corners of a field.

Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: