An article in The Atlantic explores how memes based on an XKCD comic illustrate the growth of fluff in the scientific literature. This is hardly a new notion in this blog (ahem, like here and here and here and here and here and, sheesh, probably another five or ten posts). Basically, those who determine whether an academic has sufficient value are apparently limited in their evaluative abilities to integer math learned before they hit double digits in age. “Me count papers, more GOOOOD, few BAAAAD.”
But it was these lines from the article that inspired GG to get his grumpy up and respond:
“Everyone recognizes it’s a hamster-in-a-wheel situation, and we are all hamsters,” says Anirban Maitra, a physician and scientific director at MD Anderson Cancer Center…. Maitra has built a successful career by running in the publication wheel—his own bibliography now includes more than 300 publications—but he says he has no idea how to fix the system’s flaws. In fact, none of the scientists I talked with could think of a realistic solution. If science has become a punch line, then we haven’t yet figured out how to get rid of the setup.Scientific Publishing Is a Joke, The Atlantic
Really? Nobody can think of a solution? They sure aren’t trying hard.
1) Limit the number of publications that can be considered for promotion and tenure. Five seems a fair number. GG thought Harvard had instituted this some years ago, so maybe there is some federal law against it, but this would dampen the enthusiasm for publication.
2) The contribution of the academic seeking promotion to each paper must be spelled out clearly. Providing lab space, owning an instrument, buying snacks don’t count.
3) You can only submit a paper after having reviewed at least 2 other papers. “Olé” reviews (“This paper is great, publish as is”) don’t count. Wanna push out 20 papers this year? Enjoy reading those 40 other manuscripts.
4) Develop a repository for the failed projects that were externally funded. A lot of the crap in the literature comes from the “requirement” (that, by the ways, usually is not official) that every grant must produce publications. So archive the stuff that really doesn’t amount to a hill of beans so that the granting agencies can see it rather than coerce some of us to review it and reject it multiple times before it settles into its publication grave somewhere.
5) End author lists that are longer than movie credits. Either you were really contributing to the analysis being done, or you didn’t. GG has serious doubts that all 150 authors on a Nature paper could even state clearly what was in the paper, let alone describe their essential contribution (“I brought cookies!”).
6) Failure should be an option. Want to try something risky but with a potential high reward? This should not be a career-killer.
And that is just pondering this for a few minutes. GG suspects you all have some equally good ideas and would love to hear about them in the comments…
To be clear, we are talking scientific publication. And to save some of you time, there isn’t a lot new here, but the trends are looking to collide sooner rather than later.
What does GG mean by a publication apocalypse? Basically the end of any meaningful evaluation of publications; we are heading rather rapidly into a blizzard of material with no vetting or meaningful review. While there are those who think this will be the most democratic way to distribute science, GG would rather point them to how equally unmoderated blizzards of material have led to minor problems in the political sphere like, oh, insurrection based largely on falsehoods.
What are the trends that are facing the whole concept of journals?
- open access upon publication
- high levels of publication for tenure, promotion and funding
- preprint servers
- junk journals
- reviewer fatigue
- expansion of research into more of academia
Several of these interact in poisonous ways. The perception that faculty must publish more and more frequently to satisfy promotion and tenure committees or funding agencies leads to lots of manuscripts circulating around, all of which need to be reviewed, thus leading to reviewer fatigue. Open access demands on journals are probably putting journals fully into the realm of vanity publishers: their only source of income will be what authors pay. This in turn restricts the support journals can provide to editors and reviewers. Researchers who aren’t flush with funds (for instance, many summer intern programs or honors thesis writers or advanced degree recipients who were supported on teaching assistantships) will be forced to either limit their findings to preprint servers lacking any review or junk journals that claim peer review despite lacking it. The increasing pressure on traditional journal publishers will slow the path to publication, making those junk journals more attractive.Read More…
Some years back many of us saw an opportunity to make electronic publication of science more than just a limitless pile of page images. We could have interactivity: the deeper and richer datasets and models being created and interpreted could be more fully presented to readers who could interrogate the data directly rather than having to, say, download raw tables and replicate a bunch of work before being able to see if looking at the data a bit differently yielded a different interpretation.
And so some of us demonstrated such tricks by embedding interactive material into pdfs of scientific papers. While some of this was the trivial use of layers or embedding a movie, other examples were more sophisticated. These typically relied on Flash.
Well, as you no doubt know, Flash is no more. Adobe, like the rest of the world, has moved on to HTML5 tricks and beyond. But what was stranded–entirely and apparently irrevocably–are animations embedded in pdfs. There is no tool for embedding html5 in pdfs. There is no scaled-down Flash viewer that works with pdfs. Nope, all you get now from those pdfs is a message that Flash is dead and a worthless link to an FAQ online that doesn’t even consider the possibility that you got there from trying to look at a pdf.
GG had hoped that there would be some substitute. The underlying language of Flash is in fact open source. Adobe wouldn’t just utterly kill off a major capability of their software, would they? Well, the answer is now crystal clear: it is “yes”. And what is more, there is no real substitute out there (the closest are epubs, which are not particularly well supported, especially when it comes to real interactivity).
In an online-always world, maybe you shrug and convert over to html 5. Which, realistically, means you only can access a document live as it will depend on a host of other files to fully become a document. And after the misadventures with Adobe and Flash, just how future proof do you think html5 is going to be? So interactive scientific publication is, once more, dead. Yes, you can do a few tricks with layers and (oddly enough) embedded 3-d models still work (an odd thing to carry forward). But anything more than that is dead and gone.
GG has a couple of papers using these tools that are now very difficult to read; one had even won an award.
All this was unnecessary. Macromedia and then Adobe could have knifed off the troublesome hooks that Flash had developed into the operating system and returned it to a local, sandbox-friendly tool. Apple sounded the alarm when they prevented Flash from running on iOS many years ago. But they fell prey to the same fantasy of write-once-deploy-everywhere that has been a goal for many, from Bill Gates’s versions of Basic to the initial promises of Java and on to Flash (frankly, Python is showing signs of the same lack of hubris).
So we raise a proud middle finger to Adobe and its carefree disregard for its users. You showed us a path forward and then took it away. And so many of us now wish the same oblivion for you.
With the nation’s focus on early voting, voter suppression, the Supreme Court, not to mention “COVID, COVID, COVID”, some recent actions of the Trump administration are not getting the airing they deserve.
First up is best summarized by the lead in a NY Times article:
The Trump administration has recently removed the chief scientist at the National Oceanic and Atmospheric Administration, the nation’s premier scientific agency, installed new political staff who have questioned accepted facts about climate change and imposed stricter controls on communications at the agency.New York Times, Oct. 28, 2020
Diving down, we find that the administration has a political minder censoring emails within NOAA; any communication from the NOAA administrator could be blocked on purely political grounds. While we have seen from time to time administrations try to block public statements from agency personnel, this level of intervention appears to be unprecedented.
But wait, there’s more! Apparently feeling that there are too many people not sufficiently under the thumb of the Trump administration, a new executive order seeks to gut a fair part of the civil service. It is worth recalling that the civil service was created in response to the corruption of government associated with political spoils. While prior to the civil service being created the goal of politicians was to have plum jobs they could offer to their supporters, in this case it appears that the idea is that competence at a job be second to loyalty to the party line of the chief executive. Why does this matter? Let’s listen in to recently resigned chairman of the Federal Salary Council and lifelong Republican Ron Sanders:
It’s absolutely critical because of the complexity of that world — the laws, the rules, the regulations, the scientific theories, all of the things that go into public policy. Somebody has to understand that. You can’t look at the CliffsNotes and get it. You need people with deep technical expertise who are there regardless of party who provide neutral competence to whoever is in power.NPR story, 31 October 2020
With any luck at all, maybe this can be the last “War on Science” post. Maybe all of us have learned that science can help to prevent awful outcomes like we are seeing today from the novel coronavirus.
With the Black Lives Matter movement and protests came repeated demands for having “uncomfortable conversations.”
GG finds this highly insulting. Before you fire up Twitter or something, GG is not saying we should not discuss racism and the impacts of slavery or Jim Crow or economic history on Black Americans. It is the insistence on making people uncomfortable that is objectionable. You see, how you feel is how you feel; telling people how they should feel is mistaken (this is probably counseling 101). If GG says that you must be uncomfortable for a conversation to be successful, then that becomes the goal. Which, frankly, is missing the point. You want to have conversations about meaningful corrections to a university curriculum? Fine. About some appropriate reparations? Fine. You want to make corrections for white priviledge? OK. Might someone feel uncomfortable? Sure, maybe even likely, but that shouldn’t be the goal. Phrasing it as “we have to have uncomfortable conversations” will put people off. How eager is anybody to do something uncomfortable if that is the primary justification? You don’t go to the dentist because it is uncomfortable, you go because there is a problem that needs solving. Say we need meaningful conversations and yeah, we might anticipate it could be unpleasant, but with a more important goal in sight, we can overcome a feeling of dread and choose to engage. But don’t frame the conversation on how participants must feel.
Of course, this same damaging mindset is present on the right, and in spades. How often do we see glee at “owning the libs” or making liberals angry or unhappy? Really? This is what we want to accomplish is simply making fellow citizens feel badly? Sometimes this is stupid (GG recalls some GOP fans trashing some Keurig coffee makers over the coffee maker pulling ads from a Fox show. Why they thought destroying their own property was going to somehow anger liberals is a mystery to GG). The problem as GG sees it is not that people end up being angry or hurt–that can be the result of necessary discussions on topics that have been hard to resolve–it is that a lot of us try to make other people angry or hurt.
How much of this is the anonymity of social media or the disintegration of social bonds like church or service organizations is unclear. How much is thirst for celebrity by doing something especially outlandish? In doing fieldwork, GG has many times met people with very different political views but being there, in the flesh, we had some interesting civil discussions. It’s a lot harder to sit across a table from somebody and treat them as dirt.
There are a lot of big divisions out there. We don’t agree on the problems the country faces. We sometimes don’t agree on what is actually occurring. None of this will improve if all our goals amount to making others feel bad.
It constantly amazes GG just how many weird ways there are to work the scientific publication system. Years ago, for instance, an author, fearing the review of a colleague, added that colleague as a co-author in the submitted version of a paper and then removed that person when the paper was published. This avoided having that colleague review the paper. Variations on this scheme exist. The flip side is no better: leaving a contributor out because of an attempt to gain more credit for some work than is deserved. Being forced to add a lab director or other powerful purpose is another obvious abuse of the system. There are games in citations, too. But now GG learns that there is another way people have been gaming the system, not with authorship, but through the acknowledgements section.
The acknowledgements? Where petty blandishments are collected to salve the wounds of irate reviewers (“We thank reviewer Smith for damning us all to hell, which improved this paper”)? A section that screams “common courtesy”? Really? The section that you’d only read if you were looking for your own name? That section?
But yes, apparently so. Some editors use the acknowledgements to strike reviewers off their to-invite list, so the acknowledgements can be used to discretely vet the choice of reviewers. Apparently some journals actually require that those acknowledged be contacted so they can have their names removed if desired. Honestly it just seems like people are paying way too much attention to this. GG has done field work where we would often acknowledge the assistance of land managers and property owners in giving access. You are really going to want contact info for all those folks?
The paper linked above (and here, from a pointer at Retraction Watch) describes a falling out among authors that led to a paper published with a former author being acknowledged. After a long series of messages, the journal removed the acknowledgement. This seems to imply that people do read those acknowledgements.
Honestly, the level of petty signaling and maneuvering that goes on is reaching unanticipated heights. Before long we’ll discover that a paper’s titled spoken backwards says “John Smith is a jerk” or something like that. How hard is it for scientists to behave like nice people?
8/30: clarified that Vietnam War deaths refers to American casulties.
Science really has never been the ultimate decider of government rules and regulations; that role is played by government officials who have to interpret the information scientists provide. The U.S. Weather Service does not issue evacuation orders, local and state officials do. The U.S. Geological Survey does not decide which buildings can and cannot be built on an active fault, state and local officials do. And so on and so forth.
This is what makes the attack on science over the past 3.5+ years so pernicious. Previous administrations had made decisions in opposition to what the science advisors had suggested–but at least (in general) they had heard that advice and decided other considerations were more important. If the Trump administration wanted to overrule the scientific advice they were getting, there was nothing to prevent that. What is impressive is that they couldn’t stand the optics of ignoring scientific advice (sometimes to the point where they did the exact opposite of what the science was suggesting).
And so it should not surprise anyone that after purging the EPA, redrawing weather maps, bullying career employees to not say what should be said, leaving scientific positions open, preventing the USGS from commenting on trends beyond 2040, turning down NOAA assistance in a major disaster, demanding that all studies that used private data be withdrawn from use in making policy that the administration would turn its attention back to public health. We’ve now seen two “emergency use authorizations” from the Food and Drug Administration that look very much like political decisions, and then a change in advice from the Center for Disease Control about COVID-19 testing that appears to reflect the president’s desire to see less testing more than a scientifically justifiable adjustment to testing advice (the justification provided would have suggested that the website text needed to be fixed, which has not happened). Toss in placing a partisan activist into the FDA as spokesman for the agency (a move that gratefully lasted all of 11 days) and it is clear that, once again, the administration simply does not want there to be any actual scientific advice to creep into public discourse.
This kind of approach seeks to disguise the utter failure of American public health to corral the coronavirus, but its impact is to make the attempts to control the disease less effective. Yes, there were always going to be deaths, there was always going to be economic disruption, and yes, some other countries botched things too. (GG can find eight others having a higher per capita death rate over the entire pandemic: in decreasing order, San Marino, Belgium, Spain, the United Kingdom, Chile, Italy, Sweden and Brazil, but of these only Brazil and Chile–and the US–are seeing death rates continue to rise at rates of 1 or more newly dead per 100,000 per week). But the magnitude and duration of failure in the U.S. is probably only really matched by Brazil.
GG has argued that you ignore science at your own peril. We as a nation are now paying the price for that mindset, which again is different from consciously choosing to overrule science. We will never know just how things might have gone had science not been sidelined throughout the government, but it is hard to imagine things being worse and easy to believe they would have been better. American deaths in the Vietnam War played a large role in ending the Johnson administration, forever tarring that presidency with failures that have diminished its successes. We’re now past three Vietnam War’s worth of deaths; will this administration pay any price for this?
OK, so GG was asked about how to write a scientific paper after suggesting how you might read one. And, well, there are whole books and fully researched articles and stuff like that out there. But you know, that’s entry level scientific writing. Here’s the real advanced stuff!
- Don’t start writing until all the work is complete! Why waste your time scribbling down things while you are in the midst of career-defining research. Every second counts! Plenty of time to write once you know the outcome; this way you avoid writing something you ditch later on.
- Don’t keep notes. You don’t want some snooping competitor to find out what you’ve been doing. Best to just know in your heart all those algorithmic choices you’ve made. If you must write stuff down, be sure to use a code that isn’t described anywhere. No, Leonardo’s mirror writing tricks are passé now…
- Don’t give talks before submitting for publication. Again, no point in tipping your hand. And after all, are those ninnies who always snipe at you really going to give constructive criticism?
- Rely on the one computer script that rules them all. Sure, every time you want to adjust a parameter in your statistical analysis or simulation or what not, you overwrite that Python script while leaving plots and data files laying about from older versions of the script, but you are keeping track mentally, right?
- Delete intermediate results. Less clutter the better, so deleting all that intermediate stuff is a great way to retain organization on your computer drive. Also makes it far easier to ignore demands from reviewers down the road (oops! that stuff is gone).
- Ditch the background stuff. Blah blah blah, yeah, standing on the shoulders of giants. More like standing on the shoulders of gnats, rightly squished under your boot heel. Why waste the space in Nature or Science with drivel just meant to appease friends of old farts?
- Find the big story. Originally you were learning the age of some speciation event but you realized the data is predicting the next great earthquake, well, forget that trivia you started with, even though you don’t really know much about earthquakes, your speciation dataset will support you through this.
- Maximize surprises in the text. You’ve held this great work close to the vest so far–so you want the big reveal at the end, just as in all those murder mystery books. So throw in a few red herrings, make sure the abstract is misleading or obtuse so that the reader will be totally blinded by the brilliance at the end of the paper.
- Minimize figures. I mean who really can’t make their own figure by downloading the binary supplemental information from your website, whose URL somehow got changed after publication. Real scientists read and digest huge data tables for breakfast!
- The more jargon the better. Just who the hell do these readers think they are? Only true peers who know the particulars of your vocabulary of polysyllabic ultra-germanic mashups are worthy of receiving the wisdom of your work. Let’s keep the riff-raff out!
- Maximize inferences. Sure, you just studied one rock from some corner of your garage, but that rock–that rock–tells the entire history of the solar system. Your conclusions should take full advantage of these insights.
- Be a press master. Your college’s PR department loves to have click bait out there to prove that Big State U is really on top of the cutting edge of science. Feed their needs–why yes, this is revolutionary, and yes, it does show that Einstein was wrong and oh, of course this points the way to a cancer cure, too. No point in having this seminal work get overlooked.
And there you are, ready to take on the world. GG suggests having an off-shore bank account and maybe a spare passport just in case your brilliance attracts the attention of some authorities…
A widespread talking point on the right has been that a “Deep State” would act to block actions from President Trump because such a body would be some liberal fifth column. This view has always flown in the face of the reality of large organizations. First off, there is organizational inertia: doing anything requires a lot of effort to get things moving in some direction, whether conservative or liberal or just plain middle-of-the-road. So any change of direction will be slow to descend through an organization.
But another aspect of organizations is, “keep your head down.” Which is summarizing a strategy on order to keep your job and avoid doing something that might attract attention. And in this environment, managers lower down in the system will take cues from those at the top and shift priorities to more closely align with the current leadership. And so it is no surprise that the New York Times has reported that a significant amount of the editing and revising and reversing of scientific information within the government has been done by civil service employees instead of political hires. (This appears to have been inspired by an inspector general’s report on such behavior at the EPA). The extent of this behavior is kind of amazing. Of course, at the high end is the business of chastising the Birmingham, Alabama, weather service office for, um, trying to suppress panic from an incorrect assertion from the President all the way down to suppressing the near automatic statement of grant support or professional affiliation in a published paper. The New York Times article is particularly of interest owing to the reproductions of numerous internal documents documenting these actions. It also mentions an example from the previous administration, which tends to illustrate the universality of behaviors within a bureaucracy.
Strong objections to pharmaceutical companies preventing publication of some of their funded drug studies has led to rules that help to allow all such studies to see the light of day. The suppression of scientific work undertaken with federal grants or by federally employed scientists deserves no less an effort to assure its presentation within scientific journals. Yes, the science might skew away from that administration’s current goals, but tough. If you aren’t ready other bad news and judge it on its merits, you really shouldn’t be involved in setting government policy….
A nice essay in the New York Times is slugged (online) as “How You Should Read Coronavirus Studies, or Any Science Paper.” It details a bit of the history of scientific prose and notes the familiar standard background/methods/results/discussion structure of such papers. But it doesn’t actually tell you how to read such a paper. In fact, about the most specific advice given is to find authorities on social media and have them inform you. Which is fine for really hot button stuff like coronavirus research, but might leave you at sea in most other fields.
So, putting aside the lingo specific to that field, the ineptitude of authors’ prose, the gutting of the paper by dismissal of key parts to supplemental materials…just how do you approach a scientific paper? Well, no worries, here is GG’s guide. And a hint: don’t just read these papers front-to-back…Read More…