“Is Science Bullsh*t?”
John Oliver’s answer was an emphatic “no,” but he went on to show how much of science reporting is–nothing surprising to those here, one presumes, but it is worth going to YouTube (or HBO) and seeing how he does a nice job showing how off target this reporting can get: (as usual with HBO these days, some language not for sensitive ears).
(GG is most fond of Oliver showing a morning show host saying that you should just wait for the study that agrees with what you want to do, a behavior Oliver declares is not science, it is religion). Now while this is immensely entertaining and hits the target rather solidly, Oliver notes that replication studies are really important, but is that really the case?
Now to be clear, a scientific result needs to be one that can be verified and/or reproduced [much earth science would be verification as we don’t reproduce a mountain range]. But do we want to fund a minimum of two sets of experiments for every study? As many have noted, part of the problem is publication bias, so encouraging the publication of negative results will help to put the one study with positive results in better context and might head off the need for any attempt at replication.
But often there are not other attempts to address the same phenomenon; what then? GG recalls what his physics roommate once told him, that the Nobel prize is only awarded for work that has been the basis for subsequent advances. How is this relevant? In the process of building on the yet-to-be-Nobel-prize-winning research, other researchers have (presumably) verified that these results were valid. If they were fake or a fluke, subsequent research would have presumably gone nowhere. Is this the most cost-effective way forward? Since we really only need to replicate the work that can serve as the basis for other advances, perhaps so. Maybe the most rational way to allocate resources is to wait to see somebody propose to build on the results of some study rather than simply fund replication studies willy-nilly. Yes, it is possible (nay, likely) that many such studies will not achieve their original goals–but they should test the studies they are built upon.
Perceptive pessimists will say, well, a study could assume the original study’s conclusions and somehow reach even more irrelevant conclusions without identifying possible flaws in the original study. The answer is to encourage some level of replication in funding the new studies; most scientists will only try to build on something once they’ve confirmed it for themselves anyways. The greater risk, in GG’s view, is that if we are flooded with promising but erroneous science, we will spend all our time debunking the bad results and will not be poised to build on the stuff that is actually robust. The answer here, GG thinks, is not to encourage opening the floodgates by publishing every last scientific experiment as some would argue, but to engage in honest and thorough peer review so that we winnow out a lot of the bad results before anybody is tempted to try to build on their results. Good peer review = fewer replication studies.