We’re Number 2!…Did we try harder?

A few months ago, US News and World Reports came out with a global ranking of universities by discipline (as opposed to their more usual ranking of schools overall or individual departments) and much to the surprise of many, GG’s home at the University of Colorado (CU) came in number 2 globally in geoscience.  Since the only department with anything like that name is Geological Sciences (also GG’s home), the school linked the department to this ranking. Woo-hoo! Take that, MIT and Stanford and all you other snooty places! This so impressed other departments that asked for speakers from our department.

Now the reality is a lot more complex.  On one hand, this is an appropriate (and long overdue) recognition of the strength in earth sciences at CU, but on the other, it isn’t quite the resounding endorsement of Geological Sciences that it might seem. This certainly is not a ranking you want to use to decide where to go to school (potential students can look at a page GG has at his academic website on how you might want to look at rankings), but it might be a ranking of use if you wanted to aid a high-achieving research program. Because this stuff gets to be so prominent and wielded as a club for getting resources, it might be worth looking behind the rankings to see what is going on. It might not be what you think, and it probably isn’t telling you where the best science is per se.

There are two parts to this: what at CU contributed to this ranking, and what exactly is it that US News and World Reports looks at.

OK, so what constitutes geoscience for the ranking?  They wrote “Geosciences is the study of Earth, from its structure to the history of its formation. Topics in the field include oceanography, petroleum geology, geology, geochemistry, geophysics and climatology.” So certainly all the faculty in Geological Sciences contribute, but then so do nearly all the tenure-track faculty and researchers associated with INSTAAR and CIRES, both of which are research institutes within CU. Faculty who are part of the Geophysics PhD Program should count too. And given the definition, presumably faculty in ATOC are all contributors (their webpage suggests as much).

When you pile that together, you are looking at people in Geological Sciences, Atmospheric and Oceanic Sciences, Physics, Geography, Chemistry and Biochemistry, Civil Engineering, Aerospace Engineering, and two biology departments (MCDB and EEB). And yet, you only are seeing part of the picture, because CIRES and, to a lesser extent, INSTAAR employ a large number of PhD-level scientists who are soft-money researchers. (Here “a large number” means more than the number of faculty we’ve just covered).

Earth Science at CU is big. If we pulled everybody together into a School of Earth Science, there would be no mystery whatsoever about this ranking. So you can see that Geological Sciences itself is only a small part of the whole.

Big, by itself might not buy you much, depending on what you measure.  So what are the criteria?

US News lists global reputation (12.5%), regional reputation (12.5%), publications (15%), citations/paper (10%), total citations (15%), number of publications among the 10% most cited (15%), percentage of total publications that are among the 10% most cited (10%) and international collaboration (10%).  So who is citing you is 50% of the score, with a squishy reputation score making up 25% (reputation plays a far larger role in the ranking of departments) and the last quarter is total pubs and collaboration.  The time frame is 2010-2014. Many of these are actually normalized by using the log of the values, then all are converted to a z score, which is the number of standard deviations the school’s values are above the mean (negative values if below). Some of these are pretty vague, so let’s dive down a bit more…

Reputation is a beauty contest from a survey. Although US News claims 50,000 respondents to the survey, the survey conductors themselves (Thomson Reuters, aka Clarivate Analytics) only  claimed something over 7,000 responses. How many were actually familiar with geoscience is unclear (probably not more than 1 in 10). Such surveys are always backwards looking and biased by success of institutions in placing their graduates in places where they answer these surveys. This is, objectively, worthless.

“Publications” is, amazingly, just counting total publications in “high-quality, impactful journals”. Not normalized by anything.  You got more people, you get more pubs.  Same with total citations, no control for size of the school. So 30% of the measure is, really, size of the institution. And if you have a lot of soft money people who have to publish that much more, well, off you go.

Arguably another 35% is a bit more skewed towards good work (citations/paper, number of highly cited papers, percent of papers that were highly cited). But this is relying on citation numbers.  It isn’t clear how minutely they tried to differentiate between very slow moving fields and hot fields; probably not at all as their goals are to reward more impactful players and GG guesses they think that where the literature churns the most must be the place where the impacts are being made [this is quite arguable]. Looking at the topic list provided in Science Citation Index, the top 3 papers and 6 of the top 10 are climate science (2 address, oddly enough, Precambrian geology in China, one hydrology, and the other oceanography). Schools with big climate change programs will beat out those with, say, a heavy paleontological emphasis.

To summarize, 25% is beauty, 30% is largely size, 35% is emphasis in a hot field. That last 10% (international) is the fraction of papers coauthored with international scientists (controlled by country: Luxembourgish researchers are quite likely to collaborate outside the country; US researchers, less so). CU probably did poorly in the beauty contest precisely because CU hides the size of its earth science programs by sprinkling them across many departments and two schools. But on size, well, CIRES is enormous and nearly entirely in the geoscience focus area. Toss in that much of CIRES’s and INSTAAR’s work is in climate science, one of the hottest fields around, and it becomes clear just how CU became number 2.  In fact, the surprise isn’t CU being number 2, it is Caltech, which is far smaller, being number 1. GG’s guess is that the beauty contest part helped Caltech a lot just as the size measures worked against it to some degree.

So number 2 really means being a heavyweight in the more active parts of the field. And if taken that way, this ranking probably isn’t far off the mark for CU as a whole.

 

Tags:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: