Sinking ratings

So the New York Times is warning us that Harvard is no longer the top-rated source of science in the world and the rest of US institutions are falling behind. They point to the Leiden rankings as being an appropriate measure.

GG has dived into the ranking mess a few times, in part because it is important to potential graduate students (so some of that analysis is on GG’s page for prospective students), and this has come up from time to time in this blog too). The trend that NYT noticed has been ongoing for quite awhile. Basically, China is swamping everybody with the production of scientific research. This is of course producing a panicked reaction in some quarters. Just what is going on?

Is Science Doomed?

Recently we’ve been focusing on damage to the scientific enterprise being done by the current US administration (datasets vanishing, long term studies canceled, capricious grant revocations). But maybe that’s just noise in a broader decline that might prove unstoppable.

Nature reported on a study published in PLOS Biology that documents an explosion in low quality papers exploiting a huge, open dataset (in this case, the National Health and Nutrition Examination Survey (NHANES) dataset). And explosion it is, as such papers have grown by a factor of 50 or so in the past few years. These authors note two issues: one is the use of a simple analysis approach that is unsophisticated and not really appropriate, the other is the use of odd subsets of the data in identifying correlations. IN both cases it seems most likely that the combination of simple scripts to look for correlations along with AI codes like ChatGPT have opened the door for a blizzard of papers that are kind of industrial-scale p-hacking. It is hard to imagine that this will be limited to this one dataset, or indeed to one field of science. And this is no surprise to those of you reading this blog…

This is on top of existing problems, like publication of incremental science, overwhelmed reviewers, endless proposal writing, and scientific fraud. The fundamental problem is the incentive structure for scientists, especially academic scientists. In academic science, all too often it is numbers of papers, and in too many places it is the perceived quality of journals, that determines tenure or promotion. Toss in the increasing difficulty in getting funding and the growth in low-quality papers is easy to understand.

Now a lot of that are problems for scientists; but how about science itself? As with earlier calls to end peer-review, accepting a flood of derivative and crank-turning AI papers drowns out the few thoughtful papers that presumably are out there somewhere. So there is the needle-in-the-haystack problem. And as AI gets better at this, odds are good that the flood of marginal or sub-marginal papers will increase more. Given the current tendency of AIs to hallucinate, an increased use of these tools might create whole alternate universes of fictional papers with fictional results. Toss in the increasing levels of fraud and at some point the utility of the literature simple crumbles under all these pressures.

As practiced today, science very much relies on the ethical behavior of scientists. At some critical level, when some percentage of the literature is fake or trivial or even irrelevant, it isn’t hard to see the literature being ignored. At that point, we cease to build on the shoulders of predecessors; instead, we continually rediscover the same things. Or the possibility of finding some papers in the ocean of dreck that support the new paper you’d want to write. It would be easy to anticipate the growth of insular groups uninterested in interacting with the broader community. The idea of a “scientific consensus” would fade away.

So in the end, it comes down to how new scientists are confirmed into scientific community. It really needs to be a process that rewards real advances over publication numbers. Absent such a change, it is hard to see science become a sewer of whatever will get people hired and tenured.

Science paper genres

First a caveat: there is a literature on how science works. GG doesn’t delve into that literature, so some of this may be nonsense. But it is nonsense of a scientist and this leads in an odd direction. You’ve been warned…

This might surprise some folks (probably not many who read this blog), but not all science papers (or “articles”) are the same. There are four distinct types of science papers in GG’s zoo, and it can be helpful to be aware of this as you encounter them. [OK, missed a fifth caught by a commenter…so exercise to the reader is to fit it in].

The most common paper is the one you envision from high school science, the paper where some data was gathered and analyzed and shared. These papers are the building blocks of science; call them brick papers. They are rarely the highly cited papers, instead contributing another brick in the massive edifice of science.

Another relatively common paper is the overview paper. These tend to congregate in specific journals with “review” in their name as well as various book volumes where the contribution is anticipated to be more long lived in usage. These can be really difficult to assemble and write (the Decade of North American Geology, or DNAG, volumes of the late 1980s are a prime example). These attempt to pull together all those individual bricks together. The result is some combination of creating a bigger picture and identifying points of controversy. Such papers don’t generally solve any conflicts, instead typically falling back on “more research is needed.” These papers are often very highly cited as they allow a later researcher to plop in a citation to a review paper instead of a mind-numbing list of 20 or 30 papers that might be relevant (and might be worth citing so that potential reviewers aren’t feeling left out). These can help guide the science going forward.

A related genre are synthesis papers. You might think these are simple overview papers, but in a way these are to overview papers as editorials are to news reporting. There is a slant there that the author(s) pursue. Conflicting observations are shot down, supporting evidence is propped up, and a distinct conclusion is reached. These too can be widely cited if their chief conclusion proves to be popular. Or, sometimes, they are highly cited as a strawman quickly demolished in later papers in pursuit of completeness. In either case, how they deal with conflicting evidence can clearly identify the strongest points to be reexamined.

Finally there are papers that propose a different view of the field altogether. Call them alternative view papers, or maybe dissenting papers. Some will emerge from a rich literature review, so in a sense a synthesis paper; others emerge from a small set of observations, typically of a new type, but what they share is an expansion of viewpoints to include a perspective absent from the literature to that point. J. Tuzo Wilson’s 1965 paper on transform faults is an excellent example (indeed, many early plate tectonics papers land in this category). Although these usually emerge from some dissatisfaction with existing theory, they are far enough from the mainstream that both points of support and objection are utterly different and so open new lines of research.

Too Many Channels

Most GG’s age remember a time when there were three TV networks and maybe one or two local stations. From these folks you might hear “thousands of channels and nothing to watch”! Well, it isn’t just TV where this is true…and in this other case it maybe is more damaging. GG speaks of social media.

First, though, let’s expand on the term to include many collaborative environments because they work, in the end, very much like small social networks.

Recall in the early 19th century you had two means of communication: oral face-to-face and writing a letter. (A book is really just a lot of letters strung together). By 1960 we were up to phone calls and, if you really needed written urgency, telegrams. (Or FTD, if you wanted to say it with flowers). Since about 1993, options have mushroomed.

So GG was talking with a group of geoscientists lamenting that they weren’t sure how to best inform colleagues of their group’s activities. They were old-school enough to be contemplating email flavors or a society’s connected community site or a newsletter or a web page. So we’re talking pretty old school here, and yet the choices can be difficult. Pick poorly and you don’t get the message across, so not an empty decision.

But it can get worse as generations cross. Instagram and Twitter/X might work better than, say, FaceBook or LinkedIn, or some might want Slack or Discord or Mastodon or a Reddit thread or even a lowly blog. Any new collaboration or membership demands a new communication tool. Open a computer and exclaim “Too many choices! Which channels have something I need to see on them?”

Of course the problem is that there are differences between these tools, and so we face some folks wanting this tool for this reason and others that tool for that reason. So this isn’t likely to go away anytime soon. But we may need a means of coping with this. There are hints of awareness in the social media development community (the interoperability of Mastodon would allow conversations from one server/interface talk to others, much as a phone call from L.A. to Boston might involved three phone companies but worked the same as calling a neighbor). But there have been hints before that went nowhere (think of Bill Gates’s crusade for “Write once, deploy everywhere” for …. Basic!).

You can almost bet that if somebody makes the equivalent of a smart interchange for all these that some media will try to break it as exclusivity is where they gain an edge. You almost wonder if, one day, the government will step in and say “Enough”.

That lone should get the powers of Silicon Valley working on this post haste…

Proof Science Doesn’t Matter…

A story from the American Geophysical Union meeting was about a couple of climate activist scientists getting shut down as they interrupted a speaker to unfurl a banner urging scientists to go in the streets and do civil disobedience to get attention to climate change. One of the scientists ended up getting fired after being removed from the meeting and having her science presentations removed. Whether or not this was an overreaction isn’t what GG is interested in here (it turns out that the speaker who was interrupted was themselves controversial and the society was worried there would be attacks directed at that speaker). Instead, in an op-ed she penned for the New York Times, the fired protester said “When scientists take action, people listen.” Is this true?

Really? How many of you were aware of these scientists chaining themselves to things in Washington DC? Can you even find news stories about this?

Wanna share just how the March for Science a few years back changed the path the Trump administration was on? Just how many scientists gained elective office after that (hint: you won’t need toes to count them up). And is it civil action by scientists that matters, or is it getting people informed about the hazards that matter? Was it Silent Spring that really mattered, or was it some sit-in by scientists somewhere that led to DDT being banned?

You want proof the opinion of scientists is not a political force at all? When doling out committee assignments, where did House Speaker McCarthy decide to ship off liar extraordinaire George Santos? Yep, the Science, Space and Technology committee. Yeah, science and scientists are pretty important players in setting the agenda politically…that’s why you want the black sheep of the flock working on it.

The title of this post is misleading, really. Science does matter; it is the (political) opinions of scientists that do not. Once you step out and say “we must do this” (as opposed to “this is what is happening and why”), you are joining the legions of petitioners to the government…and there are a lot more of them than of you.

What is Peer Review for, anyways?

Every now and then GG encounters things as he dons the different hats of a professional earth scientist that just require revisiting some aspects of the job. And so today’s topic is peer review.

First let’s dispense with the obvious: peer review is not saying “the scientific community has examined this research closely and guarantees it is free from mistakes and blunders and is a true representation of reality.” Yes, many of you are laughing, of course it isn’t! But the public often is led to think this is what it is.

Now, we do often like to say that it is a means of preventing bad science from being published. This is also untrue: with the plethora of journals out there, it is awfully hard to prevent bad science from being published somewhere. GG has had the experience of leading an author point-by-point though mistakes in their work only to have that author claim to agree but leave the same wretched mistakes in the revised manuscript which, once rejected, promptly showed up in another (clearly less discriminating) journal. At most we might prevent bad science from being published in our journal, but unless the AEs and editors are all in lockstep, even that is too strong a statement. A related and fairly uncommon (in GG’s experience) situation is a paper that fails to recognize it isn’t offering anything new; in this case, this might prevent unnecessary duplication within the literature. Of course there are the examples of deceit that we now find occasionally in the literature; these are really outside the pale and not part of the run of the mill execution of the scientific enterprise.

GG’s view is that normal peer review will usually address two things: one, whether the journal in question is appropriate, and two, if or how authors have failed to clearly make their point. There are perfectly fine works that simply are not a good fit for a journal. Ideally the editor catches these and pitches them back, but sometimes it is the reviewers. This is typically a small part of the operation for anything other than the most high profile letter journals. The second point, though, is really where peer review matters most.

Silencing Comments

A thread on an AGU bulletin board emerged demanding that an AGU journal return to allowing the practice of comments and replies. Many went so far as to call the absence of a comments policy to be an assault on science. The basic argument is that if you find an error in a paper that should be corrected, there is no easy way to point this out without a comment–the error might not itself amount to a new publication.

This is kind of an interesting conundrum. On many public-facing websites and publications, the comments section is proof of a lowest possible level of discussion with ad hominem attacks and irrelevant discussion [not here, of course!], so it is kind of amusing to see such a venue desired in the scientific community. Except, of course, the scientific comments are not off-the-cuff challenges to the intelligence of the authors, but are instead carefully written documents that point out an issue in a published article. As such, these are usually reviewed at minimum by the editor of the journal and often are sent out for mail review. So what’s not to like?

Pronouns and Papers

Over the past few years, it has become standard practice in many universities to list preferred pronouns for individuals.This allows individuals to be characterized in a manner consistent with their desires. It might be time to carry this into the scientific literature, but probably not the way you are thinking.

Consider the following from one of GG’s papers:

Some numerical experiments by O’Driscoll et al. (2009) explored concepts relevant to this hypothesis; they found that the presence of a lithospheric root will lead to… 

Jones et al.., Geosphere; February 2011; v. 7; no. 1; p. 183–201; doi: 10.1130/GES00575.1

This is pretty standard usage, yet what really is the antecedent for “they”? It happens to be O’Driscoll et al. (2009). Which is a single paper, not a group of people. This is exceptionally common usage in the literature, in that we assign to the authors the results of the paper. Part of this is avoiding personifying an inanimate object (the paper didn’t “find” anything any more than your water glass “found” the water in it). But occasionally when people change their minds, you can find some things that might look silly. For instance, it would be fair to write “Jones (1994) felt that the Isabella anomaly came from the lithosphere under the Tehachapis, while Jones et al. (2014) argued it came from under the Sierra Nevada.” Now that is the same Jones, yet the interpretations are different. And this is because the earlier paper doesn’t change meaning as the author revises interpretations. So would you really want to write the same information this way? “Jones (1994) used an early seismic array to image the Isabella anomaly. He argued it came from under the Tehachapis, while Jones et al. (2014) argued it came from under the Sierra.” This Jones fellow seems pretty slippery, no?

This might seem silly; after all, the meaning is still pretty clear; author Jones changed his mind somewhere between 1994 and 2014. But the thing is, by personalizing the paper–making the association between the author(s) and the paper so strong that they are interchangeable–we make it that much harder for readers to separate a specific product of a specific study with the individual(s) who wrote up the study. Once we release a paper into the wild, it is gone, not to be fixed by a later change of heart. It is the paper that will continue to make a claim long after its author has moved on. So maybe we should use pronouns other than “his” or “hers” and move on to “its” to make clear that it isn’t the current state of the authors that we are looking at, it is the material presented in the paper.

There would still be instances where referring to the author(s) instead of the papers might make sense. Consider this (from the same paper):

Only Bird’s (1988) specific geodynamic version of the flat slab has provided quantitative predictions at a lithospheric scale from trench to foreland; it is based on a review of the physical relations of several aspects of the hypothesis (Bird, 1984). In developing his model, Bird sought to not only produce deformation far to the east of earlier shortening…

In this case, the text in the last sentence is stepping out from the two papers cited to consider motivations driving the development of the published papers. Now this might not be fair, but it is exploring what the person was doing from the 1984 paper to the 1988 paper. There is still some ambiguity of timing, but in this case it isn’t quite the papers per se being considered but the person doing the work.

Anyways, for more straightforward examples, would the use of more generic pronouns for scientific publications be kind of annoying for us fossils? Sure, but then we’ve had to deal with personal pronouns of “they/them” that just produce cognitive dissonance as our internal English teacher lashes out from years ago. If we can deal with that, maybe we could depersonalize the presentation and discussion of science in a way that makes it easier for authors to later revise their views without seeming to be contradictory.

Whose job is it to share science?

Consider these two quotes. One is from the FAQ for Merchants of Doubt:

Many scientists think that their “real work” is in the field or the laboratory, and that communicating science in plain language is someone else’s job. We think that should change.

Merchants of Doubt website, FAQ page

The other is from a recent editorial in Science:

…[A]sking someone to be a skilled science communicator after taking one [science communication] course is like asking someone who has taken a course in chemistry to discover a novel reaction. Truly well-trained science communicators—individuals who devote their lives to helping the public understand research—deserve more respect from their research colleagues.

Tharp, It’s not as easy as it looks, Science editorial, 2021

Keep in mind that we’ve been getting increasing pressure from all sides to be our own science journalist. From the continuing expansion of the “broader impacts” component of grant applications to the “plain language” summaries in some journals to the requests from universities for documentation of reaching out to the broader community. Frankly, being an academic scientist is already challenging enough, wedging in actual research between teaching commitments and writing grant applications and recruiting and supporting graduate students, it is pretty overwhelming even before running your own science communication office.

We really need to prioritize.

Killing an Academic Punchline

An article in The Atlantic explores how memes based on an XKCD comic illustrate the growth of fluff in the scientific literature. This is hardly a new notion in this blog (ahem, like here and here and here and here and here and, sheesh, probably another five or ten posts). Basically, those who determine whether an academic has sufficient value are apparently limited in their evaluative abilities to integer math learned before they hit double digits in age. “Me count papers, more GOOOOD, few BAAAAD.”

But it was these lines from the article that inspired GG to get his grumpy up and respond:

 “Everyone recognizes it’s a hamster-in-a-wheel situation, and we are all hamsters,” says Anirban Maitra, a physician and scientific director at MD Anderson Cancer Center…. Maitra has built a successful career by running in the publication wheel—his own bibliography now includes more than 300 publications—but he says he has no idea how to fix the system’s flaws. In fact, none of the scientists I talked with could think of a realistic solution. If science has become a punch line, then we haven’t yet figured out how to get rid of the setup.

Scientific Publishing Is a Joke, The Atlantic

Really? Nobody can think of a solution? They sure aren’t trying hard.

1) Limit the number of publications that can be considered for promotion and tenure. Five seems a fair number. GG thought Harvard had instituted this some years ago, so maybe there is some federal law against it, but this would dampen the enthusiasm for publication.

2) The contribution of the academic seeking promotion to each paper must be spelled out clearly. Providing lab space, owning an instrument, buying snacks don’t count.

3) You can only submit a paper after having reviewed at least 2 other papers. “Olé” reviews (“This paper is great, publish as is”) don’t count. Wanna push out 20 papers this year? Enjoy reading those 40 other manuscripts.

4) Develop a repository for the failed projects that were externally funded. A lot of the crap in the literature comes from the “requirement” (that, by the ways, usually is not official) that every grant must produce publications. So archive the stuff that really doesn’t amount to a hill of beans so that the granting agencies can see it rather than coerce some of us to review it and reject it multiple times before it settles into its publication grave somewhere.

5) End author lists that are longer than movie credits. Either you were really contributing to the analysis being done, or you didn’t. GG has serious doubts that all 150 authors on a Nature paper could even state clearly what was in the paper, let alone describe their essential contribution (“I brought cookies!”).

6) Failure should be an option. Want to try something risky but with a potential high reward? This should not be a career-killer.

And that is just pondering this for a few minutes. GG suspects you all have some equally good ideas and would love to hear about them in the comments…