Who’s counting?

Another day, another piece in the scientific literature arguing that we need to stop counting publications and instead focus on quality-a noble goal.  The latest, in Nature, is focused more on the biomedical research literature, but the recommendations sound familiar:

Politicians must understand that job creation is not — and furthermore, should not be — a primary goal of the NIH or any other science-funding agency. Funds should be distributed on the basis of merit alone, not geopolitical considerations and interests. Institutions need to realign their mentality with their original academic mission, and reduce soft-money positions. Publishers should care less about publishing flashy stories and more about disseminating solid science. Individual scientists should emphasize excellence and rigour over stockpiling more and more papers and grants.

Quite the wish list. This paper is kind of interesting for two reasons.  One is it does make you ask, just who is counting?  And two, it claims to identify the moment that the scientific endeavor went off the rails.

OK, so who counts, and what do they count? The most impactful counters are faculty promotion committees and the external writers on tenure cases.  If they don’t care, gratuitous publication would stop. They mainly count publications (and to a lesser extent, citations) but also count grants and/or grant dollars. And the question then is, why?

This is where this particular paper gets interesting, as it suggests there was a major turning point for NIH-funded fields:

Between 1998 and 2003, the annual budget of the US National Institutes of Health (NIH) almost doubled, soaring from US$13.7 billion to $27.2 billion. This applauded increase was supposed to reward good work and reduce the time scientists spent applying for grants, but it had unintended consequences.

The infusion of funds increased the scientific workforce and the number of products marketed to scientists, but did little to boost excellence. It brought about a ‘quantity, not quality’ approach that is the antithesis of what science should be about. It fuelled an idea that academic science is a business that has to sustain constant yearly growth. This ‘businessization’ of science, with its emphasis on job creation and translatability, is undermining the freedom of ideas that allows huge, often unpredictable, progress.

The paper goes on to note the feeding frenzy that followed, with institutions expanding to try to capture some of this money and numbers of students, postdocs and faculty increasing as well, returning grant success rates to pre-increase levels or lower. (Other science funding agencies did not see an equivalent increase at this time, so there is a great natural experiment sitting out there for some social scientists). Basically, the author argues that pursuit of the almighty dollar destroyed research.

(Now, the author is being a bit naive here, as others as far back as 1965 argued that science was being diluted by becoming a comfortable profession due to government support. And GG has groused about publication counting and grantsmanship and growth of research institutions and numbers of scientists. So this isn’t as new as the author represents.)

Just why do those academic groups count pubs and dollars?  Well, in part because setting up a science lab can cost a lot of money; universities that pony up multimillion dollar startup packages are looking to get that money back from overhead that is brought in by researchers and tuition dollars of graduate students.  Those funds then support the startup for the next new hire, and so on. Breaking this cycle would change how laboratory science is done (especially in biomed). Similarly, grant agencies are scrutinized by Congress and they have to go before congressional committees to argue for their budgets.  Sharing the results of a blue-sky science breakthrough are difficulty enough to make to an academic audience; for a roomful of politicians, more tractable measures are desirable. Basically, if you have money coming from the government, it will be hard to avoid some crappy proxies for success.

What of the solution suggested? Would politicians accept spending money that might only flow to a few congressional districts? Where it might not produce jobs? And where it might not produce any results whatsoever? Where it might not produce a workforce useful to the broader economy? How would we pare back overgrown science departments at unworthy institutions? Who determines the unworthy institutions?

Yes, we need to move away from the orgy of incremental science that surrounds us, but it isn’t clear that the path advocated is a workable one. We can and should fix the metrics used within academia that incentivize faculty in a bad way, but the money side of things is going to continue to distort science.


4 responses to “Who’s counting?”

  1. Paul Braterman says :

    The problem seems intractable. Universities will expand to absorb the funds available, administrations will be properly concerned that a new hire will be able to attract funding commensurate with their research plans, and hiring and tenure committees will be full of people uninformed about a candidate’s speciality and therefore forced to rely on surrogates.

    And yet we do manage to make things worse than they have to be. I was recently asked, as external reviewer, whether the number of a candidate’s publications was in line with expectation for promotion, I have heard of one very highly regarded university department demanding that members bring in a certain level of funding if they want to keep their jobs (thank you,Thatcher, for abolishing tenure) and it is grotesque that it is more praiseworthy to achieve a certain scientific result by spending £100,000 of public money than by spending £10,000.

    Liked by 1 person

    • cjonescu says :

      “And yet we do manage to make things worse than they have to be.” This is key. While that Nature piece pleads for (unlikely) changes everywhere, we all can insist that letters soliciting external evaluations do NOT ask about numbers, do NOT ask about funding, but DO ask about the scientific impact of a colleague’s work. And in writing letters, we can similarly ignore similarly stupid requests (here at CU, there is text that the university demands be part of letters requesting evaluation letters–we then make clear to referees what we really want). Do all that and it becomes easier to demand changes in the behavior of others.

      Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: