Bishop Hill What the public needs to know about GCMs
May 7, 2014 Climate: Models Anthony has a completely brilliant comment from Robert Brown about the ensemble of climate models and the truth about them that is never explained to the public:
...until the people doing “statistics” on the output of the GCMs come to their senses and stop treating each GCM as if it is an independent and identically distributed sample drawn from a distribution of perfectly written GCM codes plus unknown but unbiased internal errors — which is precisely what AR5 does, as is explicitly acknowledged in section 9.2 in precisely two paragraphs hidden neatly in the middle that more or less add up to “all of the `confidence’ given the estimates listed at the beginning of chapter 9 is basically human opinion bullshit, not something that can be backed up by any sort of axiomatically correct statistical analysis” — the public will be safely protected from any “dangerous” knowledge of the ongoing failure of the GCMs to actually predict or hindcast anything at all particularly accurately outside of the reference interval.
Reader Comments (52)
RG Brown has been promoted from humble commenter to lead author of a new thread surely more often than anyone in the history of the climate blogosphere. I always gain much from his insights and clarity. Hopping over to WUWT now ...
Done. Superb.
GWPF said this recently
http://www.thegwpf.org/models-behaving-badly/
Can anyone answer another of my simple questions? Are GCMs intended to provide predictions of future climate or global mean temp? Are they meant to provide illustrations depending on some emission scenario? Are the results really expected to be useful, and if so on what terms?
Oh, and if you propose to excuse, say, a pause in temp increase by some hitherto unappreciated mechanism, should you be required to have that mechanism in your model either by design or as an emergent result?
As I suggested years ago when I first saw a GCM ensemble, the appropriate population from which uncertainties should be calculated includes all model runs from all participants, perhaps - perhaps - excluding those runs that went iterating off into the wilderness and were junked.
There is NO WAY that the stats should be done on a cherry picked subset. That is rather elementary.
The madness got even worse when Gavin Schmidt advanced a nonsensical argument on his blog that the real Gaussian spread of model output was greater than the IPCC assumed and this larger spread overlapped the observational spread so the models were not invalidated after all. The logical extension of this argument is that the worse the models are, the better they are. However this rather obvious logic flaw didn't stop Santer et al (30+ of the dolts) from gettting a paper published using this Schmidt argument. Annan who vigourously defended the Schmidt/Santer argument at the time, later reverted to Bayesian methods as the only sensible method to use: Something very obvious to the rest of us and mentioned in several prior papers on climate model comparisons. Of course it's only pseudo-Bayesian with enormous inbuilt bias but that's another story.
It is apparent that despite some of these climate clods having PhDs in maths they don't seem to comprehend even the very basics of stats: Such as assuming only a handful of data or models can be a priori assumed to be Gaussian, using a linear trend to represent a nonlinear process, starting a warming trend line at a point they persistently tell us was artificially cooled by aerosols and oh so many more...
rgb is a hero.
[Why does his name somehow make me think of the video circuitry in colour TV's?]
Strong stuff and about time too.
I'm sure that Dr. Richard betts would want to comment on this.
What is perhaps most laughable about the GCM models is that, of the 100+ that are now being bandied about, none actually agrees with any other, though all are deemed to be correct, thus an “average” can be used to frighten the sheeple. Utterly absurd, but the MSM lap it up.
What is perhaps most laughable about the GCM models is that, of the 100+ that are now being bandied about, none actually agrees with any other, though all are deemed to be correct, thus an “average” can be used to frighten the sheeple. Utterly absurd, but the MSM lap it up.
(Sorry about the tautology – just as bad as PIN numbers. Was not paying the proper attention, AND forgot to login.)
(Sorry about the tautology – just as bad as PIN numbers. Was not paying the proper attention, AND forgot to login.)
(And I have no idea why my posts are duplicating themselves! Well, it could be like rabbits, I suppose...)
"As I suggested years ago when I first saw a GCM ensemble"
Is "ensemble" the right collective noun for the outputs of a group of climate models?
It sounds far too sensible and respectable.
Dr Brown contributes some brilliant stuff at Ant's regularly.
Strong stuff and about time too.
I'm sure that Dr. Richard betts would want to comment on this.
May 7, 2014 at 1:20 PM | Unregistered CommenterDon Keille
You gest* of course, Don. Perhaps he could persuade slygo slingo to come up with one of her nonsensical PR sentences.
[jest? BH]
Sorry to rain on this parade but what distinguishes these GCMs from climate sensitivity models with which BH has had a love in for the last year or so?
Is that a question for Nic Lewis, our host or all others on 'BH' Dung? (And can we lose the tendentious 'with which BH has had a love in' in future dialogue, please. This isn't a matter of romantic attachment but, as I perceive it, profound differences in how grounded the different kinds of models are as far as real-world observations are concerned. But, just as with Robert Brown on CGMs, I have much to learn.)
“If you live with models for 10 to 20 years, you start to believe in them…A model is such a fascinating toy that you fall in love with your creation.”
- Freeman Dyson (At the age of five he calculated the number of atoms in the sun.)
Richard you are not and never will be the arbiter of what I choose to say and how I say it so take a tablet.
I'm not the arbiter but I can politely ask for the question not to be loaded tendentiously. Nobody has fallen in love with climate sensitivity models but take the sarcasm out and the question is an interesting one.
Richard
One tablet was obviously not enough, an overdose maybe?
I've made the point. I'm not going descend to ad hom.
To get back to 'my' point:
What makes climate sensitivity models more acceptable on BH than GCMs? I suspect it is the fact that recent models claim that sensitivity is less than thought earlier.
A very smart cookie is Dr. RGB. The comment I made at WUWT was:
"Again, it goes to show what a colossal waste of time, energy, money and manpower the whole CAGW scam has been – and continues to be.
The climate system is chaotic. Only a complete imbecile would try to model it. Surely?"
Anyway, that was my simplified take. It's all so obvious to me that it's not worth wasting good neuron time on. I post these things on Facebook, for what it's worth. I think I've made a few converts along the way.
Dung
Calm down, FFS! You are in danger of becoming as tedious as MydogSpartAlec. You know my views on CO2 and whether "heretics" should be given airtime (for want of a better word) but continually sniping at Andrew turns people off.
It's his blog; he can set any rules he likes
Richard
Ditto your pontificating every time Dung opens his mouth. Why don't you two kiss and make up or take your ongoing spat into a quiet corner of the Discussion page where we can leave the pair of you in peace to scratch each others eyes out! <:-(
Mike Jackson:
I think that's the important point.
Dung
If I may answer your question - these new sensitivity calcs (not models) were putatively observationally based. Lewis did recalculations of older work using up to date observations and some error corrections. There is still an inbuilt assumption that a certain portion of the warming seen can be attributable to mankind which is based on the idea that GCM's can separate out manmade warming from the background noise - which we now know they can't. Given that caveat these calcs can only ever be upper bound estimates and the upper bounds are found to be lower than before. Of course the originators of these methods no longer like them because they don't show alarm with up to date data but it is important to do the calc anyway so that the out of date, more alarmist, calculations are no longer touted about.
Again there is an argument there about the statistical methods - if you have sufficient real data then you can use frequentist methods, otherwise it has to be Bayesian. Lewis argues using objective versus subjective but frankly it falls on deaf ears.
If i may suggest we use the word models only to describe the numerical models and not just any method or notion that pops into anyones head. Mosher is to blame for the mixup here: He still touts the satellite data as being based on a model which is nonsense either derived from listening to other people who know crap all about models or is deliberately designed to obfuscate and irritate.
This comment from Dr. Brown was quite penetrating:
" all have a number of vested interests in there being an emergency, because without an emergency the US government might fund two or even three distinct efforts to write a functioning climate model, but they’d never fund forty or fifty such efforts. It is in nobody’s best interests in this group to admit outsiders — all of those groups have grad students they need to place, jobs they need to have materialize for the ones that won’t continue in research, and themselves depend on not antagonizing their friends and colleagues."
And so the insular party goes on and on.
I suspect it is the fact that recent models claim that sensitivity is less than thought earlier.
May 7, 2014 at 4:45 PM | Registered CommenterDung
I agree with that. From one point of view, if sensitivity models say that warming is nothing to worry about, then they are great - they could potentially result in policy changes - even if they are fundamentally rubbish ("rigorous argument from inapplicable assumptions").
I might not have used the phrase "love in" but sensitivity models certainly seem to have had a cordial reception on BH over the past year or so - for more cordial than their cousins the GCMs.
But I think they are all from the same family and to be treated with equal suspicion until there is reason to think otherwise. (I'll confirm what I say when I have finished reading "A Sensitive Matter".)
The GCMs have the misfortune that their failed predictions are already apparent. The quality of the predictions of climate sensitivity models will only be apparent to our descendents several generations from now. By which time the "Global Warming Delusion" will be just an additional chapter added to Mackay's book.
One thing to note is that Dr Brown asserts that multiple poor models cannot be better than a single poor model so it is wasted effort. But as Wm Briggs has also pointed out - with weather models indeed they can be. We don't really know why but it happens. However to be included in the ensemble each individual model has to have a basic validation test and hindcasting is a necessary but not sufficient test. Alas hindcasting is touted as sufficient validation by climateers which is either rank stupidity or a base lie. There is no evidence yet that crap models do any better when combined.
Judging by the IPCC efforts the only models that ever seem to come close to reality are those with no positive feedback. Bear in mind that the ensemble does include a base model with 1.1K of additional warming per doubling which is what the base theory tells us so the main thing demonstrably wrong with the IPCC spread calc is in not admitting that the spread is nowhere near gaussian. It is really a skew with a mode at 1.1C, given that we really don't know if feedback is negative or positive or net zero.
The paragraph you highlighted Bish is the whole crux of it. Basically we are supposed to trust a "wet finger in the air".
However, taking a more serious note for a second, it means that the models haven't been verified for use as there is no standard that they can be checked against. If the models were modular as he is saying an analysis could be performed showing cumulative error effects, much like what NPL (National Physical Laboratories) often advocate.
Also if we look at it in a financial way, the risk associated with using this advice would be very large indeed as there is not way to assess impacts using any statistical techniques.
Basically the models cannot be audited. And since RGB has many years experience with "physical models" he is essentially making an "expert" comment.
I currently work in aircraft software and have also built particle in cell models so I have an idea about how things should be done and how they are checked against standards. I don't want to be dramatic but could a case of criminal negligence be brought against the Met Office for using models to dictate flood plain policy, especially if CGM's were used?
I'd rather the Met Office didn't get this type of heat, as it's a bit reactive rather than proactive. It's just that when I read the RGB quote something struck a chord.
Feel free one and all to dismiss this as idle speculation though. It's the audit trail aspect that is really the main focus for me.
May 7, 2014 at 5:31 PM | Registered Commenter Mike Jackson
I think I only made one comment about the Bish which was to to object that 'the whole of my post was removed' when some of it was entirely relevant. On sensitivity models I think the great majority of BH posters were praising them at the same time as condemning GCMs and at that time I think that Martin A, Rhoda and myself were the only ones to dissent. In the case of Richard I have only responded to his attacks on me, I have not made any comments on any other posts he made. Indeed I would prefer it if he adopted the same position because it is obvious we do not see eye to eye on issues that are not science related. I am not continually sniping at Andrew and would never do so but I reserve the right to complain about (perceived) injustices.
JamesG,
In Numerical Weather Forecasts an ensemble is not an average of different models, as it seems to be the case in climatology.
In NWF an ensemble is several runs of the same model, with the same physics and the same parameterizations, but with small perturbations in the initial conditions. It is very useful because it gives an idea of the stability of the forecast. Typically you may see a narrow range for a few days and increasing dispersion after the fifth day, which is very telling.
An average of different models is just nonsense.
See an example for the weather here
"In NWF an ensemble is several runs of the same model, with the same physics and the same parameterizations, but with small perturbations in the initial conditions."
The ECMWF started doing these while I was studying at Reading, exactly as you describe. The problem is you know when weather is predicatble or not in the UK. Depressions tracking in from the Atlantic and the forecasts can be terrible within 24 hours, while a high pressure system sitting over the country can easily mean predictably the same weather for a week. The problem with the latter is it seemed impossible to predict with any accuracy when these stable systems would break down, so yet another poor forecast was due at the end.
" Climate models have a sound physical basis and mature, domain-specific software development processes."
"Engineering the Software for Understanding Climate Change", Steve M. Easterbrook Timothy C. Johns
I mostly stayed clear of the BH CS
love-instushie, but I think I did make one comment where I stated my gut feeling was that CS was only 0.25C per doubling at the most (and probably negligible after allowing some time for feedbacks to kick-in) so for the record, I am also with Dung on this. But it is gut feeling based on climate history, not any expertise in complex maths and stats.I think that Martin's summary at 5:53 PM is spot on.
I am reminded of comment I read somewhere by Roy Spencer (irrrc), or it may have been Dick Lindzen; and I paraphrase:
(And I have no idea why my posts are duplicating themselves! Well, it could be like rabbits, I suppose...)
May 7, 2014 at 1:48 PM | Unregistered CommenterRadical Rodent
Or, perhaps wire coat hangers?
"One thing to note is that Dr Brown asserts that multiple poor models cannot be better than a single poor model so it is wasted effort. But as Wm Briggs has also pointed out - with weather models indeed they can be. We don't really know why but it happens."
Dr. Brown is correct. Briggs (and Nick Stokes) is wrong. And we do know the reason why this happens sometime and it is very simple. It is the same reason that a stopped clock has the correct time twice a day. Nothing more than coincidence. If I say that I am clairvoyant and can tell you what number between one and ten that will be drawn out of a hat, it should be obvious that I will be correct 10% of the time. According to Briggs line of thought, this is proof that I am clairvoyant 10% of the time, which is of course bunk.
Sometimes there are only a small range of possible answers. Just because someone has a scientific procedure that can get a somewhat close answer means nothing.
The thing that is confusing is that model output is not actual data, it is only model output. Briggs' argument is valid for accurately measured real data. It does not work for "guessed at" data from a computer model.
Averaging model output appears (to me, at least) as profitable as the pooling of ignorance. Ib fact, I can see little difference in either.
Interesting discussion on an important subject. As an interested outsider I am convinced that Dr Brown is right in his assessment, but I wish that he had actually quoted verbatim the two paragraphs in 9.2 he refers to instead of only providing his own interpretation. Maybe one of the learned participants here can correct this error by quoting the original wording.
I would not expect Dr Betts to participate. His appearance is sparse and probably choreographed.
Dr Brown speaks with authority on the things we have all suspected for some time. Why does the scientific establishment in this country approve of such practices?
Gary: wire coat hangers are not known to be rodents. However, if you know otherwise...
(Always end an admonishment on a cheerful note.)
Bruce Cunningham
I suggest you research multi-model ensemble forecasts. It is not an idea that the IPCC thought up; rather it is commonplace.
Patagon
a) The main GCM's don't differ in the basic physics by enough to make such a distinction between different models and different runs of the same model. Most of the discrepancy is due to different inputs.
b) Multi-models using different techniques can also be useful in fluild flow sims.
Note that I already made the point that there has to be a certain criteria for inclusion and that must be based on how close they are to reality. You won't get the true answer if they are all biased in the wrong direction. So Briggs and Brown are just talking past each other. Note that there is no real statistics involved since their is seldom enough models/runs to produce a statistical model.
Dung:
I don't agree with this statement. I wasn't praising them and I don't remember others praising them. Please also note that this disagreement does not comprise an attack on Dung as a person, it's merely a difference about what he's saying. Where the person concerned felt the need to respond with insults I deduced that he knew that his earlier wording had been weak. It's always better not to descend to ad hom for this reason.
I've left this comment till now because this thread was meant to be about RG Brown's brilliant and very well communicated takedown of the GCMs. As our host made clear in his title - smart man that he is - it is vital that the general public understands the utter improbability that these massive pieces of software will ever tell us anything useful about future climate, particularly in such a manner that world-changing policies should be based on them. That is utterly idiotic. To divert in any way from that message seems to me very unwise.
The question of how we should view sensitivity models in the light of the disaster area that is GCMs is a separate but I think interesting one and I hope it is covered in future BH threads.
I am getting confused with the terminology in the side-debate. I thought that sensitiviity (or at least TCR) is something that emerges from the GCMs. So I am not sure what these "sensitivity models" are. Are these the energy-balance models that people like Nic Lewis use? If this is so, I do not see why someone can simultaneously criticise a GCM and praise an energy-balance model, because they are different ways of looking at the same problem.
should be "cannot"
That makes more sense diog. But I still wouldn't use 'praise' but couch it in terms of usefulness or reliability. So
But I think, as I've already said, that further discussion of the matter should be on another thread, if necessary a user-defined one.
JamesG
Someone else commented, " you can't average nonsense and get an accurate answer." Your contention that multi-model ensembles are established practice is meaningless because, as I pointed out, it only works with real data or validated model output. IPCC model runs have been shown to be wildly incorrect, yet somehow they are still relied on for guidance. Averaging wrong answers does not give a correct one.
I suggest you read Dr. Brown's article again. He is correct, the IPCC is wrong.
UK Met Office publication "Warming A guide to climate change", 2011