Every now and then GG encounters things as he dons the different hats of a professional earth scientist that just require revisiting some aspects of the job. And so today’s topic is peer review.
First let’s dispense with the obvious: peer review is not saying “the scientific community has examined this research closely and guarantees it is free from mistakes and blunders and is a true representation of reality.” Yes, many of you are laughing, of course it isn’t! But the public often is led to think this is what it is.
Now, we do often like to say that it is a means of preventing bad science from being published. This is also untrue: with the plethora of journals out there, it is awfully hard to prevent bad science from being published somewhere. GG has had the experience of leading an author point-by-point though mistakes in their work only to have that author claim to agree but leave the same wretched mistakes in the revised manuscript which, once rejected, promptly showed up in another (clearly less discriminating) journal. At most we might prevent bad science from being published in our journal, but unless the AEs and editors are all in lockstep, even that is too strong a statement. A related and fairly uncommon (in GG’s experience) situation is a paper that fails to recognize it isn’t offering anything new; in this case, this might prevent unnecessary duplication within the literature. Of course there are the examples of deceit that we now find occasionally in the literature; these are really outside the pale and not part of the run of the mill execution of the scientific enterprise.
GG’s view is that normal peer review will usually address two things: one, whether the journal in question is appropriate, and two, if or how authors have failed to clearly make their point. There are perfectly fine works that simply are not a good fit for a journal. Ideally the editor catches these and pitches them back, but sometimes it is the reviewers. This is typically a small part of the operation for anything other than the most high profile letter journals. The second point, though, is really where peer review matters most.Read More…
A thread on an AGU bulletin board emerged demanding that an AGU journal return to allowing the practice of comments and replies. Many went so far as to call the absence of a comments policy to be an assault on science. The basic argument is that if you find an error in a paper that should be corrected, there is no easy way to point this out without a comment–the error might not itself amount to a new publication.
This is kind of an interesting conundrum. On many public-facing websites and publications, the comments section is proof of a lowest possible level of discussion with ad hominem attacks and irrelevant discussion [not here, of course!], so it is kind of amusing to see such a venue desired in the scientific community. Except, of course, the scientific comments are not off-the-cuff challenges to the intelligence of the authors, but are instead carefully written documents that point out an issue in a published article. As such, these are usually reviewed at minimum by the editor of the journal and often are sent out for mail review. So what’s not to like?Read More…
Over the past few years, it has become standard practice in many universities to list preferred pronouns for individuals.This allows individuals to be characterized in a manner consistent with their desires. It might be time to carry this into the scientific literature, but probably not the way you are thinking.
Consider the following from one of GG’s papers:
Some numerical experiments by O’Driscoll et al. (2009) explored concepts relevant to this hypothesis; they found that the presence of a lithospheric root will lead to…Jones et al.., Geosphere; February 2011; v. 7; no. 1; p. 183–201; doi: 10.1130/GES00575.1
This is pretty standard usage, yet what really is the antecedent for “they”? It happens to be O’Driscoll et al. (2009). Which is a single paper, not a group of people. This is exceptionally common usage in the literature, in that we assign to the authors the results of the paper. Part of this is avoiding personifying an inanimate object (the paper didn’t “find” anything any more than your water glass “found” the water in it). But occasionally when people change their minds, you can find some things that might look silly. For instance, it would be fair to write “Jones (1994) felt that the Isabella anomaly came from the lithosphere under the Tehachapis, while Jones et al. (2014) argued it came from under the Sierra Nevada.” Now that is the same Jones, yet the interpretations are different. And this is because the earlier paper doesn’t change meaning as the author revises interpretations. So would you really want to write the same information this way? “Jones (1994) used an early seismic array to image the Isabella anomaly. He argued it came from under the Tehachapis, while Jones et al. (2014) argued it came from under the Sierra.” This Jones fellow seems pretty slippery, no?
This might seem silly; after all, the meaning is still pretty clear; author Jones changed his mind somewhere between 1994 and 2014. But the thing is, by personalizing the paper–making the association between the author(s) and the paper so strong that they are interchangeable–we make it that much harder for readers to separate a specific product of a specific study with the individual(s) who wrote up the study. Once we release a paper into the wild, it is gone, not to be fixed by a later change of heart. It is the paper that will continue to make a claim long after its author has moved on. So maybe we should use pronouns other than “his” or “hers” and move on to “its” to make clear that it isn’t the current state of the authors that we are looking at, it is the material presented in the paper.
There would still be instances where referring to the author(s) instead of the papers might make sense. Consider this (from the same paper):
Only Bird’s (1988) specific geodynamic version of the flat slab has provided quantitative predictions at a lithospheric scale from trench to foreland; it is based on a review of the physical relations of several aspects of the hypothesis (Bird, 1984). In developing his model, Bird sought to not only produce deformation far to the east of earlier shortening…
In this case, the text in the last sentence is stepping out from the two papers cited to consider motivations driving the development of the published papers. Now this might not be fair, but it is exploring what the person was doing from the 1984 paper to the 1988 paper. There is still some ambiguity of timing, but in this case it isn’t quite the papers per se being considered but the person doing the work.
Anyways, for more straightforward examples, would the use of more generic pronouns for scientific publications be kind of annoying for us fossils? Sure, but then we’ve had to deal with personal pronouns of “they/them” that just produce cognitive dissonance as our internal English teacher lashes out from years ago. If we can deal with that, maybe we could depersonalize the presentation and discussion of science in a way that makes it easier for authors to later revise their views without seeming to be contradictory.
Consider these two quotes. One is from the FAQ for Merchants of Doubt:
Many scientists think that their “real work” is in the field or the laboratory, and that communicating science in plain language is someone else’s job. We think that should change.Merchants of Doubt website, FAQ page
The other is from a recent editorial in Science:
…[A]sking someone to be a skilled science communicator after taking one [science communication] course is like asking someone who has taken a course in chemistry to discover a novel reaction. Truly well-trained science communicators—individuals who devote their lives to helping the public understand research—deserve more respect from their research colleagues.Tharp, It’s not as easy as it looks, Science editorial, 2021
Keep in mind that we’ve been getting increasing pressure from all sides to be our own science journalist. From the continuing expansion of the “broader impacts” component of grant applications to the “plain language” summaries in some journals to the requests from universities for documentation of reaching out to the broader community. Frankly, being an academic scientist is already challenging enough, wedging in actual research between teaching commitments and writing grant applications and recruiting and supporting graduate students, it is pretty overwhelming even before running your own science communication office.
We really need to prioritize.Read More…
An article in The Atlantic explores how memes based on an XKCD comic illustrate the growth of fluff in the scientific literature. This is hardly a new notion in this blog (ahem, like here and here and here and here and here and, sheesh, probably another five or ten posts). Basically, those who determine whether an academic has sufficient value are apparently limited in their evaluative abilities to integer math learned before they hit double digits in age. “Me count papers, more GOOOOD, few BAAAAD.”
But it was these lines from the article that inspired GG to get his grumpy up and respond:
“Everyone recognizes it’s a hamster-in-a-wheel situation, and we are all hamsters,” says Anirban Maitra, a physician and scientific director at MD Anderson Cancer Center…. Maitra has built a successful career by running in the publication wheel—his own bibliography now includes more than 300 publications—but he says he has no idea how to fix the system’s flaws. In fact, none of the scientists I talked with could think of a realistic solution. If science has become a punch line, then we haven’t yet figured out how to get rid of the setup.Scientific Publishing Is a Joke, The Atlantic
Really? Nobody can think of a solution? They sure aren’t trying hard.
1) Limit the number of publications that can be considered for promotion and tenure. Five seems a fair number. GG thought Harvard had instituted this some years ago, so maybe there is some federal law against it, but this would dampen the enthusiasm for publication.
2) The contribution of the academic seeking promotion to each paper must be spelled out clearly. Providing lab space, owning an instrument, buying snacks don’t count.
3) You can only submit a paper after having reviewed at least 2 other papers. “Olé” reviews (“This paper is great, publish as is”) don’t count. Wanna push out 20 papers this year? Enjoy reading those 40 other manuscripts.
4) Develop a repository for the failed projects that were externally funded. A lot of the crap in the literature comes from the “requirement” (that, by the ways, usually is not official) that every grant must produce publications. So archive the stuff that really doesn’t amount to a hill of beans so that the granting agencies can see it rather than coerce some of us to review it and reject it multiple times before it settles into its publication grave somewhere.
5) End author lists that are longer than movie credits. Either you were really contributing to the analysis being done, or you didn’t. GG has serious doubts that all 150 authors on a Nature paper could even state clearly what was in the paper, let alone describe their essential contribution (“I brought cookies!”).
6) Failure should be an option. Want to try something risky but with a potential high reward? This should not be a career-killer.
And that is just pondering this for a few minutes. GG suspects you all have some equally good ideas and would love to hear about them in the comments…
To be clear, we are talking scientific publication. And to save some of you time, there isn’t a lot new here, but the trends are looking to collide sooner rather than later.
What does GG mean by a publication apocalypse? Basically the end of any meaningful evaluation of publications; we are heading rather rapidly into a blizzard of material with no vetting or meaningful review. While there are those who think this will be the most democratic way to distribute science, GG would rather point them to how equally unmoderated blizzards of material have led to minor problems in the political sphere like, oh, insurrection based largely on falsehoods.
What are the trends that are facing the whole concept of journals?
- open access upon publication
- high levels of publication for tenure, promotion and funding
- preprint servers
- junk journals
- reviewer fatigue
- expansion of research into more of academia
Several of these interact in poisonous ways. The perception that faculty must publish more and more frequently to satisfy promotion and tenure committees or funding agencies leads to lots of manuscripts circulating around, all of which need to be reviewed, thus leading to reviewer fatigue. Open access demands on journals are probably putting journals fully into the realm of vanity publishers: their only source of income will be what authors pay. This in turn restricts the support journals can provide to editors and reviewers. Researchers who aren’t flush with funds (for instance, many summer intern programs or honors thesis writers or advanced degree recipients who were supported on teaching assistantships) will be forced to either limit their findings to preprint servers lacking any review or junk journals that claim peer review despite lacking it. The increasing pressure on traditional journal publishers will slow the path to publication, making those junk journals more attractive.Read More…
Some years back many of us saw an opportunity to make electronic publication of science more than just a limitless pile of page images. We could have interactivity: the deeper and richer datasets and models being created and interpreted could be more fully presented to readers who could interrogate the data directly rather than having to, say, download raw tables and replicate a bunch of work before being able to see if looking at the data a bit differently yielded a different interpretation.
And so some of us demonstrated such tricks by embedding interactive material into pdfs of scientific papers. While some of this was the trivial use of layers or embedding a movie, other examples were more sophisticated. These typically relied on Flash.
Well, as you no doubt know, Flash is no more. Adobe, like the rest of the world, has moved on to HTML5 tricks and beyond. But what was stranded–entirely and apparently irrevocably–are animations embedded in pdfs. There is no tool for embedding html5 in pdfs. There is no scaled-down Flash viewer that works with pdfs. Nope, all you get now from those pdfs is a message that Flash is dead and a worthless link to an FAQ online that doesn’t even consider the possibility that you got there from trying to look at a pdf.
GG had hoped that there would be some substitute. The underlying language of Flash is in fact open source. Adobe wouldn’t just utterly kill off a major capability of their software, would they? Well, the answer is now crystal clear: it is “yes”. And what is more, there is no real substitute out there (the closest are epubs, which are not particularly well supported, especially when it comes to real interactivity).
In an online-always world, maybe you shrug and convert over to html 5. Which, realistically, means you only can access a document live as it will depend on a host of other files to fully become a document. And after the misadventures with Adobe and Flash, just how future proof do you think html5 is going to be? So interactive scientific publication is, once more, dead. Yes, you can do a few tricks with layers and (oddly enough) embedded 3-d models still work (an odd thing to carry forward). But anything more than that is dead and gone.
GG has a couple of papers using these tools that are now very difficult to read; one had even won an award.
All this was unnecessary. Macromedia and then Adobe could have knifed off the troublesome hooks that Flash had developed into the operating system and returned it to a local, sandbox-friendly tool. Apple sounded the alarm when they prevented Flash from running on iOS many years ago. But they fell prey to the same fantasy of write-once-deploy-everywhere that has been a goal for many, from Bill Gates’s versions of Basic to the initial promises of Java and on to Flash (frankly, Python is showing signs of the same lack of hubris).
So we raise a proud middle finger to Adobe and its carefree disregard for its users. You showed us a path forward and then took it away. And so many of us now wish the same oblivion for you.
With the nation’s focus on early voting, voter suppression, the Supreme Court, not to mention “COVID, COVID, COVID”, some recent actions of the Trump administration are not getting the airing they deserve.
First up is best summarized by the lead in a NY Times article:
The Trump administration has recently removed the chief scientist at the National Oceanic and Atmospheric Administration, the nation’s premier scientific agency, installed new political staff who have questioned accepted facts about climate change and imposed stricter controls on communications at the agency.New York Times, Oct. 28, 2020
Diving down, we find that the administration has a political minder censoring emails within NOAA; any communication from the NOAA administrator could be blocked on purely political grounds. While we have seen from time to time administrations try to block public statements from agency personnel, this level of intervention appears to be unprecedented.
But wait, there’s more! Apparently feeling that there are too many people not sufficiently under the thumb of the Trump administration, a new executive order seeks to gut a fair part of the civil service. It is worth recalling that the civil service was created in response to the corruption of government associated with political spoils. While prior to the civil service being created the goal of politicians was to have plum jobs they could offer to their supporters, in this case it appears that the idea is that competence at a job be second to loyalty to the party line of the chief executive. Why does this matter? Let’s listen in to recently resigned chairman of the Federal Salary Council and lifelong Republican Ron Sanders:
It’s absolutely critical because of the complexity of that world — the laws, the rules, the regulations, the scientific theories, all of the things that go into public policy. Somebody has to understand that. You can’t look at the CliffsNotes and get it. You need people with deep technical expertise who are there regardless of party who provide neutral competence to whoever is in power.NPR story, 31 October 2020
With any luck at all, maybe this can be the last “War on Science” post. Maybe all of us have learned that science can help to prevent awful outcomes like we are seeing today from the novel coronavirus.
With the Black Lives Matter movement and protests came repeated demands for having “uncomfortable conversations.”
GG finds this highly insulting. Before you fire up Twitter or something, GG is not saying we should not discuss racism and the impacts of slavery or Jim Crow or economic history on Black Americans. It is the insistence on making people uncomfortable that is objectionable. You see, how you feel is how you feel; telling people how they should feel is mistaken (this is probably counseling 101). If GG says that you must be uncomfortable for a conversation to be successful, then that becomes the goal. Which, frankly, is missing the point. You want to have conversations about meaningful corrections to a university curriculum? Fine. About some appropriate reparations? Fine. You want to make corrections for white priviledge? OK. Might someone feel uncomfortable? Sure, maybe even likely, but that shouldn’t be the goal. Phrasing it as “we have to have uncomfortable conversations” will put people off. How eager is anybody to do something uncomfortable if that is the primary justification? You don’t go to the dentist because it is uncomfortable, you go because there is a problem that needs solving. Say we need meaningful conversations and yeah, we might anticipate it could be unpleasant, but with a more important goal in sight, we can overcome a feeling of dread and choose to engage. But don’t frame the conversation on how participants must feel.
Of course, this same damaging mindset is present on the right, and in spades. How often do we see glee at “owning the libs” or making liberals angry or unhappy? Really? This is what we want to accomplish is simply making fellow citizens feel badly? Sometimes this is stupid (GG recalls some GOP fans trashing some Keurig coffee makers over the coffee maker pulling ads from a Fox show. Why they thought destroying their own property was going to somehow anger liberals is a mystery to GG). The problem as GG sees it is not that people end up being angry or hurt–that can be the result of necessary discussions on topics that have been hard to resolve–it is that a lot of us try to make other people angry or hurt.
How much of this is the anonymity of social media or the disintegration of social bonds like church or service organizations is unclear. How much is thirst for celebrity by doing something especially outlandish? In doing fieldwork, GG has many times met people with very different political views but being there, in the flesh, we had some interesting civil discussions. It’s a lot harder to sit across a table from somebody and treat them as dirt.
There are a lot of big divisions out there. We don’t agree on the problems the country faces. We sometimes don’t agree on what is actually occurring. None of this will improve if all our goals amount to making others feel bad.
It constantly amazes GG just how many weird ways there are to work the scientific publication system. Years ago, for instance, an author, fearing the review of a colleague, added that colleague as a co-author in the submitted version of a paper and then removed that person when the paper was published. This avoided having that colleague review the paper. Variations on this scheme exist. The flip side is no better: leaving a contributor out because of an attempt to gain more credit for some work than is deserved. Being forced to add a lab director or other powerful purpose is another obvious abuse of the system. There are games in citations, too. But now GG learns that there is another way people have been gaming the system, not with authorship, but through the acknowledgements section.
The acknowledgements? Where petty blandishments are collected to salve the wounds of irate reviewers (“We thank reviewer Smith for damning us all to hell, which improved this paper”)? A section that screams “common courtesy”? Really? The section that you’d only read if you were looking for your own name? That section?
But yes, apparently so. Some editors use the acknowledgements to strike reviewers off their to-invite list, so the acknowledgements can be used to discretely vet the choice of reviewers. Apparently some journals actually require that those acknowledged be contacted so they can have their names removed if desired. Honestly it just seems like people are paying way too much attention to this. GG has done field work where we would often acknowledge the assistance of land managers and property owners in giving access. You are really going to want contact info for all those folks?
The paper linked above (and here, from a pointer at Retraction Watch) describes a falling out among authors that led to a paper published with a former author being acknowledged. After a long series of messages, the journal removed the acknowledgement. This seems to imply that people do read those acknowledgements.
Honestly, the level of petty signaling and maneuvering that goes on is reaching unanticipated heights. Before long we’ll discover that a paper’s titled spoken backwards says “John Smith is a jerk” or something like that. How hard is it for scientists to behave like nice people?