Sunday
Jan272013
by 
Bishop Hill
Bishop Hill Bayesians on Bayes
Jan 27, 2013 Two statisticians - professors of Bayesian statistics - write in the New Yorker.
...the Bayesian approach is much less helpful when there is no consensus about what the prior probabilities should be.
That will be a bust for the climatologists then.
Reader Comments (23)
The link takes me to The New Yorker.
[Thanks - fixed]
A very good exposition of Baysiean and Frequentist statistical arguments written for the non statistician.
It has always concerned me that Climate Scientists (like many other scientists) play fast and loose with statistical procedures that they only vaguely understand and then seek to defend their incompetence when it is pointed out to them by people who do understand.
A cautionary tale
Three climate scientists and three statisticians met on a train on their way to a conference. How many tickets have you bought said one of the statisticians? Three said the climate scientists. Oh, said the statistician we only have one just watch how this works.
A little later when they heard the ticket inspector approaching the three statisticians went into the toilet together. The Inspector banged on the toilet door; one of the statisticians answered and pushed the ticket under the door. This was duly stamped and the inspector moved on. The climate scientists were very impressed.
On the way back from the conference the six met again. We now only have one ticket announced one of the climate scientists proudly to which the statisticians revealed, to the surprise of the climate scientists that they didn’t have any at all.
When they heard the inspector on his way the three climate scientists went into the toilet together. One of the statisticians then banged on the toilet door and when the climate scientists pushed their ticket out he picked it up and went into the next toilet with his two colleagues.
This only goes to show that climate scientists should be wary about using techniques learnt from statisticians that they do not fully understand.
This highlights the central problem of climate science. There is no agreement (nor any solid basis for agreement) on what constitutes normal climate variation.
Great story, Arthur - but a little punctuation would help!
A cautionary tale
Three climate scientists and three statisticians met on a train, on their way to a conference. “How many tickets have you bought?” said one of the statisticians.
“Three,” said the climate scientists.
“Oh,” said the statistician. “We only have one. Just watch how this works.”
A little later, when they heard the ticket inspector approaching, the three statisticians went into the toilet together. The Inspector banged on the toilet door; one of the statisticians answered and pushed the ticket under the door. This was duly stamped and the inspector moved on. The climate scientists were very impressed.
On the way back from the conference the six met again. “We now only have one ticket,” announced one of the climate scientists proudly. To which the statisticians revealed, to the surprise of the climate scientists, that they didn’t have any at all.
When they heard the inspector on his way, the three climate scientists went into the toilet together. One of the statisticians then banged on the toilet door, and when the climate scientists pushed their ticket out he picked it up and went into the next toilet with his two colleagues.
This only goes to show that climate scientists should be wary about using techniques learnt from statisticians that they do not fully understand.
Interesting follow-up links.
What Nate Silver Gets Wrong
Cleaning Up Science
Coldoldman: To the six ways of cleaning up science I would add a seventh. If you discover that you are working in a null field, try to earn a living with writing novels or poetry. Actually, a Dutch professor did this but forgot to inform his colleagues about his change of subject.
We none of us are experts in everything and I am certainly no expert in statistics. So whenever I had any work that involved anything beyond basic statistics, I would go and consult with the company statistician and get him to sign off anything I produced. Just the shame (not to mention cost) of getting something wrong would have been unbearable. This doesn't apply in the field of climate "science" where there is no penalty for getting the statistics wrong and big rewards for producing the "right" result with the wrong methods. Where are the peers denouncing this failed "science"? Where are all those Met Office statistical experts?
Forgive my ignorance, but I’ve never understood why the frequentist / Bayesian debate is such a big deal for climate science. I can see that it’s of importance if you’re really really interested in knowing for certain if the climate sensitivity is 2 or 3 or 4°C, or what is the average length of a piece of string.
What’s interesting though (I’d say, the only thing that’s interesting) is the likelihood of climate catastrophe. Now whether you define this as the end of humanity or a lot of annoying climate refugees knocking at the door, I just don’t see how the debate about Bayesian statistics illuminates things.
I’m not being funny, I’d really appreciate some enlightenment.
Any Popperian could tell them that even such a consensus is of no use - or rather, it is only of use if the shared conjecture as to the prior probabilities represents objective knowledge (is actually correct).
Gary Marcus is a psychologist and Ernest Davies a computer scientist (AI). I am sure they know a good deal of statistics, especially Marcus as an academic psychologist, but they are not professors of statistics, let alone of Bayesian Statistics.
Great point, Bob Layson.
It's not a terriby exciting article: in summary they criticise Nate Silver for oversimplifying the question and overstating the significance of the answer in a book written for and aimed at the general public. Wow.
Silver's chapter on Global Warming is interesting and well worth a read in my opinion. He tends to accept the greenhouse gas affect as the prime cause of warming, based on the "consensus" of scientific opinion. The consensus approach works for him in his accurate election forecasts. He also in other parts of the book comments on the err, err, and err again,process with the goal of each time improving the prediction. I believe the book's content was already set before we saw the Met issue a new 5 year prediction of a flat temperature and also Hansen coming out with a statement concerning the extended flat temperature reality. These two developments tend to be Bayesian in their approach in that "new" information came out and a revised prediction was the prudent thing to do. Silver also recommends that the scientists stay out of the political side of the issue and concentrate on the science.
Dear geoffchambers, scientists to not collect and average their best guesses and equip the answer with confidence limits; people who this have another name.
The query raised by 'geoffchambers', 11:07 AM, has caught my attention, as I warm up a bit from a walk in the rain and cold wind. I think the debate on Bayesian methods is important here, and so I'm going to have a shot at explaining why.
Bayesian methods are receiving a lot of interest these days, not least from the likes of Microsoft who have been investing a great deal in it I think (e.g. http://research.microsoft.com/en-us/people/rherb/bayes.aspx).
The more widely it becomes known, the more we must be on guard for political activists to be intent on abusing it, not least those activists who have some overlap with climate science. This is not because Bayesian methods are wrong, or should not be used. I have never used them myself, but I hold them in high theoretical regard as having had a great deal of rigorous development. On the practical side, I believe they have made substantial contributions in various areas, e.g. the area of automated decision-making involving pattern recognition in the presence of noise.
The danger lies in the techniques being mis-used by fanatics to give their visions a little more authority. The man in the street, the politician in a hectic legislature, the foundation trustee, and in fact just about everyone, are not likely to pay heed to arcane discussions of statistical techniques. But if we do, we might be able to get through to some of them whenever a major concern is raised such as the recent one about the uniform prior.
The interest Bayesian methods might have for anyone keen to spread their personal alarm about rising CO2 levels with others is that it can allow the analyst to start from a strong ‘degree of belief’ (the ‘subjective prior’) and then make dramatic claims about the future based on tiny amounts of data, or none at all for that matter. So, as the climate system continues to act as if rising CO2 levels don’t really matter very much, our alarmed one needs some other basis for projecting his or her concerns to others without throwing away all semblance of scientific probity. For example, a prior for climate sensitivity that includes a 10C rise as an equally likely option with smaller rises is quite strong all by itself. And that’s with a uniform distribution. Suppose our alarmed one plumped for say 7C as the most likely value, and with an appreciable right-hand tail at 10C. Now when a computation from observations comes in suggesting a sensitivity of 2C, and that gets folded in to produce the posterior distribution, it will still span alarming values. Not quite as alarming as before, but a whole lot better than nothing when it comes to crafting the next press release or soundbite.
Now thanks to Nic Lewis, for example but also I think in particular, the wheeze of merely using a uniform prior spanning 0 to 10C has been been exposed as being very unsatisfactory for estimating sensitivity. I suspect that this will give us considerable protection for a while yet against the use of even more dramatic priors. Given the all but unhinged condition of some people about CO2, the appearance of a prior reflecting their vivid convictions would be a sight to see. OK for a laugh, or an exercise for the student, in a grove of academe perhaps, but not at all suitable for we the public. Too many of us are, after all, too vulnerable to a scary story.
There are no precedents for such a use of Bayesian methods, as there are for using computer outputs as a spook to scare the vulnerable as demonstrated so well by the Club of Rome on resources. The effect of slick PR backed up by white coats and print-outs from a big computer was remarkable with ‘Limits to Growth’, and it has done the trick again on an even bigger scale with ‘CAGW’.
Were it not for that tough old dame, Mother Nature, tending to do very timely displays of the opposite of what the alarmed ones and their models tell us about climate, we’d be in an even bigger political pickle. If the global mean temperatures keep going down, for example, or if the polar bears keep on multiplying, if ‘global warming’ keeps on blocking our roads in the winter and boosting the ski-resorts, and if the seas and tropical keep on refusing to rise in scary ways, then to what will our alarmed ones turn to help force more of us to share their dread?
To the more distant future I presume. To where guesswork will reign supreme, and where Bayes could, as could a computer, be handy to illustrate and develop ideas from what we think we know. Given that the computer models have not been of much use for climate of late, those who would trumpet their outputs as gospel might well be feeling a bit subdued about them.
They may also be hearing good things about what might be done with the work of the Reverend Bayes. Three cheers then for Nic Lewis and others like him who can help make sure that his work is used appropriately.
But that quote means if there *is* consensus, that therefore Bayesian priors *can* be used. Therefore the alarmists must cling fanatically to their claims of "consensus", because only that paradigm of consensus allows their methods to be valid.
From the NYT article "For example, in a notorious series of experiments, Stanley Milgram showed that many people would torture a victim if they were told that it was for the good of science."
What we need to know is: How many scientists would torture a dataset if they were told it was for the good of their career/resarch department/field of research.
The physical sciences are engaged in a search for true theories and not the most probably true theories.
No general theory can be confirmed by evidence but only tested, and if untrue, refuted.
A correct conjecture may refer to probabilities but is itself not to any degree probably false.
Edward Leamer, an eminent econometrician, forecaster, and trade economist and author of the classic article "Let's Take the Con Out of Econometrics" has advocated Bayesian approaches to model specification questions (especially about whether particular variables should be included on the right-hand side of regression models). One of his interesting ideas is to do an inverse analysis where you ask "Given the data, how strong would your prior have to be to believe that there is a y% probability that the true model includes regressor X?"
I think that the inverse approach is a good way to frame many of the issues in the climate arena. For example, "Given the data, how strong would your prior on climate sensitivity have to be to believe that there is a y% chance of it being above X?" One disillusioning finding on a number of economics questions is that the data are often not strong enough to overcome even relatively bland priors pointing in different directions. My gut instinct is that the same is true for most of the important climate issues.
Statistics needs to be lobbed out altogether. It is too easy to cheat in either method. As a user and creator of models I know you only need to use percentage error versus reality. Using frequentist stats on models is just plain stupid unless you do a proper sensitivity analysis with many runs using randomised variables within their error bands. Hence only Bayesian stats can be justified at all for climate models but to use Bayesian stats you need either very good facts or real experts: A consensus of guesswork is of no value whatsoever and neither is pretending to be an expert on things that still need so much research to reduce the uncertainty bands. All of which is of course should be blindingly obvious to the modelers.
JamesG (12:07PM) methinks you are too harsh: 'Statistics needs to be lobbed out altogether'.
Statistics is defined in my view as the study of how best to test ideas against observations. This includes the design of informative experiments and other data collection plans which can help stimulate new ideas, as well as the testing of existing ones. It also includes assessing the quality and relevance of data, and the nature of any inferences which reliably be made from them.
The computer modelers can use statistically-designed experiments to explore their models' behaviour and sometimes derive response-surface representations from them that can widen the scope for verification and prediction. They can also take cognisance, if they wish, of statistical studies(*) that show no clear effect of CO2 on temperature from the historical records and perhaps compute just what kind of future behaviour of climate would be able to provide either a falsifiable forecast and an example of what we would have to observe before a statistical analysis would provide the support they might well seek.
* e.g. http://hockeyschtick.blogspot.co.uk/2012/08/new-blockbuster-paper-finds-man-made.html
and http://wattsupwiththat.com/2013/01/03/agw-bombshell-a-new-paper-shows-statistical-tests-for-global-warming-fails-to-find-statistically-significantly-anthropogenic-forcing/