Metrical Madness

And now, for your enjoyment of the ability to place your institution above others, we introduce yet another metric! (Applause, hosannas, people falling to the floor in ecstasy). And so, as is usually the case, the promotion people at the relevant universities (like GG’s) push any favorable numbers out the door, like here.

The new metric? Well, OK, technically it is now 4 years old but it seems to have gained some prominence with a recent modification: Nature Index. And just what does it measure?  It is simply counting the number of articles in a subset of “prestigious” journals over the past year affiliated with institutions.  Which journals are prestigious? You wouldn’t be wrong to say Nature journals, many of which make the cut. In earth and environmental science (where CU ranked in the top 10, much to the pleasure of the university’s promotors) the list is:

  • Earth and Planetary Science Letters (E)
  • Ecology Letters (W)
  • Environmental Science and Technology (E)
  • Geochimica et Cosmochimica Acta (E)
  • Geology (O)
  • Geophysical Research Letters (W)
  • Journal of Geophysical Research: Atmospheres (W)
  • Journal of Geophysical Research: Solid Earth (W)
  • Nature (S)*
  • Nature Climate Change (S)
  • Nature Communications (S)*
  • Nature Geoscience (S)
  • Proceedings of the National Academy of Sciences (PNAS) (O)*
  • Science (O)*
  • Science Advances (O)*
  • Water Research (E)

Letter journals are italicized and publisher in parentheses (S for Springer, E for Elsevier, W for Wiley, O for other). Asterisks are multidisciplinary journals where only those earth & environment papers were counted.

Frankly, this might be the single most useless measure out there (and GG has criticized several other metrics). Basically, the list is a beauty contest assembled by people picked by Springer.  Half of the journals here are letter journals; while these are generally worthwhile journals, several pretty well-regarded long-form solid earth journals are missing (G^3, GSA Bulletin, Geophysical Journal International, JGR: Surface Processes for starters). But that is quibbling over details.  It is the basic concept that is rotten to the core.

Just what does this metric show? It is used to see which institutions are publishing the most in these journals. And…so what? What does it mean that the Chinese Academy of Sciences is outstripping everybody in publishing earth and environment papers in these journals? Especially if there is a bounty to be had for publishing in these journals? So what if CU publishes a few more papers in these journals than UW? Who cares?

The claim is that these are the journals where scientists publish their best results.  And that is true, if by “best” you mean “highest profile” or “most newsworthy”.  As a trip through Retraction Watch will reveal, it is often the outliers that can pass through the filters at high-profile letter journals and so there are a decent number of sexy but wrong papers that make it into those journals. If by “best” you meant best supported and most thoroughly documented, then many of these are the wrong journals.

As a result, this index tells us nothing about good science, it is not telling us where this is being done, it is not even telling us where it is being published.  If you look in Web of Science for the most cited papers in environmental science and geosciences (which isn’t trivial to do, by the ways, so this could be off), of the top 10, precisely two were in one of these “prestigious” journals-0-and those 2 were long-form, not letter journals! Limiting to the better-defined geoscience, only 4 of the top 10 were published in journals from this list–again all long-form journals. It certainly provides no indication of quality for prospective students. Should this guide where money goes from funding agencies? Hardly.

As near as GG can tell, this is a cheerleading exercise for Nature-branded journals and nothing more. It is another toy ranking that university administrators can use to pretend that good things are happening at their institutions. And so you can expect somebody somewhere will use this in some stupid way to reward or punish scientists in a pretty useless way.


Trackbacks / Pingbacks

  1. Prestige | The Grumpy Geophysicist - June 19, 2018

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: