Josh BBC R4: "Everything we know is wrong"
Aug 28, 2014 Journals BBC Radio 4 produced an amazing programme this week on the problems with scientific research. Everything that has been said by sceptics about climate science was here - they even describe a 'decline effect' - how delightfully ironic. Here is the programme blurb:
Every day the newspapers carry stories of new scientific findings. There are 15 million scientists worldwide all trying to get their research published. But a disturbing fact appears if you look closely: as time goes by, many scientific findings seem to become less true than we thought. It's called the "decline effect" - and some findings even dwindle away to zero.
A highly influential paper by Dr John Ioannidis at Stanford University called "Why most published research findings are false" argues that fewer than half of scientific papers can be believed, and that the hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. He even showed that of the 49 most highly cited medical papers, only 34 had been retested and of them 41 per cent had been convincingly shown to be wrong. And yet they were still being cited.
Again and again, researchers are finding the same things, whether it's with observational studies, or even the "gold standard" Randomised Controlled Studies, whether it's medicine or economics. Nobody bothers to try to replicate most studies, and when they do try, the majority of findings don't stack up. The awkward truth is that, taken as a whole, the scientific literature is full of falsehoods.
Jolyon Jenkins reports on the factors that lie behind this. How researchers who are obliged for career reasons to produce studies that have "impact"; of small teams who produce headline-grabbing studies that are too statistically underpowered to produce meaningful results; of the way that scientists are under pressure to spin their findings and pretend that things they discovered by chance are what they were looking for in the first place. It's not exactly fraud, but it's not completely honest either. And he reports on new initiatives to go through the literature systematically trying to reproduce published findings, and of the bitter and personalised battles that can occur as a result.
You should be able to listen to the programme here.
Posted by Josh
Reader Comments (47)
I wonder how high that....
"41 per cent had been convincingly shown to be wrong"
....goes when you throw ideology into the mix.
I heard the programme a couple of days ago. I thought it was all particularly pertinent to climate "science".
"It's not exactly fraud, but it's not completely honest either."
A surprisingly accurate description of climatology, especially if you delete the "not exactly" bit and all the stuff after the word "fraud".
In a similar vain "Inside Heath" http://www.bbc.co.uk/programmes/b019dl1b/episodes/player has been investigating, "Conflicted Medicine" (three programmes)
A very interesting insight into conflicts of interest, lobbying groups and other shenanigans within medicine, but did remind me of some of the antics within Climate Change.
If you listen carefully, you can hear the sound of pennies dropping. We haven't had an Einstein class idea come out of research for over half a century, and that's despite it being funded as never before in history.
Pointman
Consider the EPA's current claim that the new CO2 regs will "save" 2100 blacks from having heart attacks. That means they can measure to 0.003% of the black population in the US (in the one thousandths) which would mean their margin of error would be in the millionths of one percent. That is about as accurate as gibberish can be.
This is the BBC, it does not allow criticism of climate "science"!
After all it is settled science, so nothing to debate!
They will wheel on an "expert" who will drivel on but it must be true if the BBC says so.
Doesn't apply to climate science because it's the religion of the chosen people, therefore it can NEVER be wrong.
Mailman
Sitting in a pub in birmingham, I overheard a university student utter the phrase,
"it's true, I read it in a newspaper". Needless to say, my jaw dropped and I turned
round and asked him if he realised what he had just said. That was 50 years ago.
I would now do the same thing if I heard someone quote from a scientific paper
in any discipline.
This week's 'Recycled Radio', on the subject of truth, is also worth a listen:
http://www.bbc.co.uk/programmes/b04f9frk
It's comedy/satire but, as you might expect from Gerald Scarfe's involvement, lands some pretty hard punches on our wonderful leaders - particularly the Bush/Blair 'truth' about weapons of mass destruction, probably the last decent scare before CAGW took over.
Haven't heard the programme, but am surprised they even thought published papers are mostly true. Scientists are, after all, working on the edge of out knowledge it shouldn't be surprising that when they test their "guesses" as Richard Feynman put it, against reality a large number of them will be wrong. There's no shame in that. The shame comes when they test their guesses against reality and it doesn't give them a clear answer and they say their guess is right.
Rearrange the following words into a well-known phrase or saying:
Chickens roost the home coming are to
BBC R4: "Everything we know is wrong"
Everything about the BBC is wrong"
ISTR reading a comment by a retired BMJ editor wherein he reckoned that, in his experience, the majority of peer-reviewed and published papers subsequently turned out to be wrong.
Despite being ironically hyperbolically titled, the program is a pretty good introduction and worth listening to. The one area that it understates as potential sources of non-replicable findings is the role of politics and ideology, especially in controversial fields such as Climate Science. It also has what I would term a kind of whining tone to it - as in "these poor scientists do it because of the terrible "publish or perish" system.
Hence the weekly news items:
'Fat is good for you.'
'Fat increases the risk of cancer.'
'You need to eat five portions of fruit/vegetables a day.'
'You need to eat SEVEN portions of fruit/vegetables a day.'
'Alcohol will kill you.'
'Wine is fine.'
'Whatever you're doing - stop it....'
But you misunderstand the nature of science. No paper is ever wrong it's just part of the process of seeking knowledge. It's unfair to hold scientists to the same standards we expect from lesser mortals. /sarc.
Saw the word "medicine" in there.
Saw the word "economics" in there.
Never saw the word "climate" in there.
If the powers that be at the BBC had an iota of self-awareness, they would surely ask themselves about the implications for they way in which they themselves report on scientific issues. The sad truth is that journalists, particularly broadcast journalists, are fundamentally lazy, and very few look beyond the press release...
QI mentioned something of this ilk a while ago.
https://www.youtube.com/watch?v=N2aFRBF7QE4
(from about 5.20)
60% of the experts' "facts" that we were told about in the first series are already thought to be untrue, just 10 years later.
So, if the study is true, apply the result to the study itself.
Did they talk much about physics? Or Chemistry? Or any of the physical sciences? I'm guessing not so much.
Yes, MJ. Odd, that. However:
Which has to include climate, as it remains one of the hottest (no pun, honest) topics extant, at the moment, and involves perhaps the largest number of scientific teams (…or does it? Are they really scientific? Who knows?), thus the findings are the least likely to be true. That conclusion, I am sure, most of us will agree with. Perhaps wholly inadvertently, but well done, BBC.Always been fascinated by science. Through the ages of 10-18 I read most of the "New Scientist" popular style magazines as they came out each month. Then I began to notice over the years that all of the extraordinary claims were not panning out. Eventually I pulled out boxes and boxes of my old dusty copies and thumbed through hundreds of them. I couldn't find any claims in any of them, a decade later, that turned out to have eventuated. I'm sure there must have been a few, but by that stage I gave up looking. Can't say I would have lost much if I'd spent the same time reading comic books instead.
But the climate crisis is settled science and cannot be questioned.
What maroons.
The best point I have read lately about the fallacy of the climate consensus is paraphrased like this:
What is wrong with a science that generates dozens of explanations for the pause but the warming has only one."
Of course we didn't mean to imply that climate science could be included. We were talking about real science .... er.... no that's not quite what I meant. What we were saying is that research carried out by people who make big money out of it .... no, that's not right either.
People like Big Pharma who stand to make megabucks by telling porkies .....
Quit while you're ahead?
I bet the BBC didn't mention this topic either:
Same-sex Adoption: Not as Harmless as Portrayed
http://www.americanthinker.com/2014/08/samesex_adoption_not_as_harmless_as_portrayed.html
It is not so much the conclusions of the paper, but the 'sloppy science', including poor sampling practice, that was allowed to escape into the public domain that is of concern, somewhat like climate 'science'.
Met Office warn of 'catastrophic' rise in temperature
Vicky Pope, Head of Climate Change Advice - Met Office
Published at 5:37PM, December 19 2008
A new study by the Met Office warns that the world could warm by more than 5C in the next 90 years, if emissions keep on rising. This would be catastrophic for the environment and for humanity. Dr Vicky Pope, Head of Climate Change Advice at the Met Office's Hadley Centre explains the science
When it comes to climate change, scientific evidence provides critical information for decision making. Because governments need to understand the consequences of choosing one strategy over another they also need to understand what will happen if targets are missed or cannot be agreed by all countries. Failures could have far-reaching consequences and so the Met Office has conducted a series of 'what if?' climate projections, to give a better understanding of what those consequences might be.
[Want the full article and website access? 30 days for just £1 etc etc]
http://www.thetimes.co.uk/tto/environment/article2144193.ece
The program gives a good summary of the issues, though it is a bit light on the ideological, political and opportunistic reasons behind some research findings. It also has a whining quality to the narrative - as if the scientists who produced problematic findings or over-hyped findings did so because of the nasty "publish or perish" system, so it wasn't really their fault.
It is refreshing to see the BBC report this. I cringe when I hear statements by their journalists/reporters like, "Scientists have found" or "Scientists say" uttered with almost breathless adoration, knowing that what follows will drive my blood pressure to precipitous heights. When will they realize that a minority, but highly vocal, sub-set of scientists are marketing a product, their ego. The report does not specifically cover climate science but the removal of science from an unjustified pedestal is a start.
(My personal experience with academic and institutional research, in the field of hydrocarbon extraction and processing, was that they tended to subscribe to the colloquialism "Never mind the quality, feel the width").
There would seem to be a sort of Gresham's Law affecting Science. Does an excess of poor scientists drive good scientists into other fields or retirement? Certainly, there's the Pew [!] study that said only 6% of American scientists are Republicans. Huff'n'Puff commented in their wisdumb that the Republican Party was 'alienating' scientists, inducing them to become Democrats.
The hiring bias in academia, however, is well-documented, resulting in (among other things) increasingly incestuous in-house day-to-day discussion and out-house "peeer review." A friend who circulated a poster (on Faecebook) citing the 6% figure missed the obvious othering involved and hopefully might have known better than to post it, had it said, "Only 6% of scientists are [insert minority of your choice here]." In Nazi Germany, of course, the poster could easily have consisted of, "There are NO Jewish scientists in Germany!" with a background trio of handsome (male) Aryans in lab coats looking pie-eyed into the rising sun.
One of my hobby horses is claiming that we are 95% sure we found a real effect. 5% of all studies will pass that miserable test just by chance, and often more if the statistics aren't the usual bell shape.
This test is just the first step in showing relevance, not the final step.
But try and tell that to the geniuses with their button for calculating the darn thing in Excel. Do-it-yourself "statistical calculations," no actual intelligence required.
These were my take-away quotes:
“there is no result that you cannot make seem plausible”
“pressure to hype your result”
“if you come up with your hypothesis after you have the data it is possible it is a chance result”
“not a lot of scientific effort is put into replication”
“one route to success is to be innovative but not clash with existing theories”
“there is less reproducibility than you would expect”
Shame they stuck to reasonably trivial science a skated past the 'biggest problem faced by mankind today"
We haven't had an Einstein class idea come out of research for over half a century, and that's despite it being funded as never before in history.
We have had ground breaking ideas in the last 50 years. Plate tectonics was 1968. The web and hypertext have transformed the world far more than Einstein every did. Genetic engineering, cloning etc. Astronomy has advanced in technical techniques to the point where we can spot planets in other systems, and the last fifty years have advanced with the background radiation discovery, the issues of missing mass, of accelerating universe etc Maths has major breakthroughs, especially in topography. Superconductors are all over the place. We have rovers on Mars.
We haven't had a pure physics "Einstein class" idea for while. But in physics they only come once a century anyway. Such things have never particularly come from funded science, and it is unreasonable to assume they ever will.
Mooloo "The web and hypertext have transformed the world far more than Einstein every did."
Would the web have existed without Einstein? I don't think so - or at least somebody else to discover the photoelectric effect and thereby get quantum theory rolling. (A pre-requisite for the development and exploitation of semiconductors.)
It is good to focus on peer-reviewed science in general, and consider whether the problems with climate science are an anomaly.
One might think science builds from small but important findings, one step at a time. It seems there is an incentive now to claim big impressive findings, and hope the evidence eventually catches up. I suspect the boomers have always been in a bit of a jam. The first half of the 20th century (just before the boomers were born) saw unprecedented findings. Since the second world war, there has been an unprecedented number of researchers chasing findings, but an apparent lack of possible big findings that are actually out there. (Of course, someone might find a cure for cancer). Supply and demand: inflation pushes up the value of findings that are actually quite paltry when seen clearly.
The (especially) fraudulent scientist Stapel was famous for publishing
and forWhat utter trash! The descriptions alone scream "fraud".
Was Climate Science ever mentioned in this respect I wonder?
Lloyd:
The cancer research establishment is every bit as rigid as any other which is buttressed with lots of money and reputations on the line.
Just as the discovery of h.pylori as the most common cause of stomach ulcers knocked the gastroenterologists back, the discovery that a virus caused many (perhaps most) cases of uterine cancer did the same.
BTW, not surprisingly, cancers of the stomach and nearby regions of the intestines were always more common with people who had ulcers. I hate to think what garbage they extrapolated from that, before the real cause was found.
Would the web have existed without Einstein?
Sorry? Of course. People made alloys before they knew about atoms. Transistors were made before we knew how they worked. (Seriously, we made them first, No-one made them based on any theoretical understanding.)
In any case, could Einstein exist without James Clerk Maxwell? To say that no modern discovery in physics matters because it wouldn't have happened without quantum theory is a sterile argument. We build on what went before.
Anyway - is this where we now post comments on the current EU lunacy - a ban on 'powerful' vacuum cleaners..?
How does this sit alongside the government's keenness for us to all drive electric cars..??
Well I am sure it is happening in all areas of science and not just climastrology.
In fact I am pretty sure that Einsteins Energy equation is not what it used to be.
I think that now
E = (about 0.6) * m * C^2.
Give or take a bit depending upon opinion...
Interesting point, Mooloo.
There are countless examples of things that were proved empirically, and used, without any theoretical basis whatsoever.
I'll have my thinking cap on about that for the forseeable. Although, it kind of echoes my comments in the GCM thread about how humans managed to design and build great structures long before the existence of computers or even advanced mathematics.
Hmmm.
Johanna,
at the risk of matching your critical thought with mush (mine): I suspect that it is impossible to design anything without some theory underlying what you're doing. The theory could be that if you do this you'll get that result. And I do think that is a theory, But I suspect what you meant by theory might be more abstract.
Provoked by one of Mosher's damned koans, I've accepted his concept that a design is a model, therefore almost anything that we do is based on a model. If that makes sense, then it is nonsense to object per se to computer modeling since it is only mechanizing what we would otherwise have done manually and which we've been doing since someone thought a rock could be made into an axe. I suspect you and I are in complete agreement on all of this.
Unless I've diluted the specificity of the things so generally that it is no longer of any real use.
I do think it's good to be wary of acting on the basis of untested models; but objecting to modeling itself is nuts.
Long, long ago, when I still had a TV, I heard a presenter announce her guest on a cooking show as a 'Nutrition Expert".
And, I kid you not, the 'Nutrition Expert' started by saying:
"20 years ago, nutrition experts were telling you that the potato was bad for you. Nowadays, we know that it contains an abundance of natural goodness...."
jferguson, Mosher's impassioned defence of models cuts no ice with me. A design is not necessarily a model, IMO, although I'm sure he wishes it was. :)
But it doesn't just apply to things that have to be designed. Take herbal medicine, for example. People discovered that willow bark, which we now know contains salicylic acid (aspirin), could be made into a useful pain reliever. That valerian can be rendered into a sedative. That putting cobwebs in a wound helps to stop bleeding. And so on. Not a jot of theory was involved.
While the line between empiricism and theory can be blurry at times, for the most it is pretty obvious. Indeed, usually theory is an ex post facto attempt to explain why things which have been observed happen.
[temporarily abandoning skeptic mode and going full cynic]
"Well, if we can't believe the scientists,
I guess we'll be forced to believe the politicians.
(state sponsored journalists are a lost cause.)"
@Doug - bit late as usual
but it's - "A highly influential paper by Dr John Ioannidis at Stanford University called "Why most published research findings are false""
forget the PR bull, then the interest for BH & us is the Climate research/IPCC findings to Governments which will/are causing widespread changes/costs to me & my grannies way of life.
as a layman i have read the ongoing "physics? Or Chemistry? Or any of the physical sciences?" disputes & agree they are intersting & relevant (not sure if i understood your point tho).
they are not trying to change the human side of the world as we know it, so usually get a pass from me as being speculative.
can you see the diffence ?
ps- my mothers family are McNeils from Barra, we might be related Bro :-)