Bishop Hill On the meaning of ensemble means
Jun 14, 2013 Climate: Models Climate: Statistics Readers have been pointing me to this comment at WUWT. It's written by a reader calling themselves rgbatduke, and it considers the mean and standard deviation of an ensemble of GCM predictions and asks whether these figures have any meaning. It concludes that they do not.
Saying that we need to wait for a certain interval in order to conclude that “the models are wrong” is dangerous and incorrect for two reasons. First — and this is a point that is stunningly ignored — there are a lot of different models out there, all supposedly built on top of physics, and yet no two of them give anywhere near the same results!
This is reflected in the graphs Monckton publishes above, where the AR5 trend line is the average over all of these models and in spite of the number of contributors the variance of the models is huge. It is also clearly evident if one publishes a “spaghetti graph” of the individual model projections (as Roy Spencer recently did in another thread) — it looks like the frayed end of a rope, not like a coherent spread around some physics supported result.
Note the implicit swindle in this graph — by forming a mean and standard deviation over model projections and then using the mean as a “most likely” projection and the variance as representative of the range of the error, one is treating the differences between the models as if they are uncorrelated random variates causing >deviation around a true mean!.
Say what?
This is such a horrendous abuse of statistics that it is difficult to know how to begin to address it. One simply wishes to bitch-slap whoever it was that assembled the graph and ensure that they never work or publish in the field of science or statistics ever again. One cannot generate an ensemble of independent and identically distributed models that have different code. One might, possibly, generate a single model that generates an ensemble of predictions by using uniform deviates (random numbers) to seed
“noise” (representing uncertainty) in the inputs.
What I’m trying to say is that the variance and mean of the “ensemble” of models is completely meaningless, statistically because the inputs do not possess the most basic properties required for a meaningful interpretation. They are not independent, their differences are not based on a random distribution of errors, there is no reason whatsoever to believe that the errors or differences are unbiased (given that the only way humans can generate unbiased anything is through the use of e.g. dice or other objectively random instruments).
So why buy into this nonsense by doing linear fits to a function — global temperature — that has never in its entire history been linear, although of course it has always been approximately smooth so one can always do a Taylor series expansion in some sufficiently small interval and get a linear term that — by the nature of Taylor series fits to nonlinear functions — is guaranteed to fail if extrapolated as higher order nonlinear terms kick in and ultimately dominate? Why even pay lip service to the notion that 
Let me repeat this. It has no meaning! It is indefensible within the theory and practice of statistical analysis. You might as well use a ouija board as the basis of claims about the future climate history as the ensemble average of different computational physical models that do not differ by truly random variations and are subject to all sorts of omitted variable, selected variable, implementation, and initialization bias. The board might give you the right answer, might not, but good luck justifying the answer it gives on some sort of rational basis.
Let’s invert this process and actually apply statistical analysis to the distribution of model results Re: the claim that they all correctly implement well-known physics. For example, if I attempt to do an a priori computation of the quantum structure of, say, a carbon atom, I might begin by solving a single electron model, treating the electron-electron interaction using the probability distribution from the single electron model to generate a spherically symmetric “density” of electrons around the nucleus, and then performing a self-consistent field theory iteration (resolving the single electron model for the new potential) until it converges. (This is known as the Hartree approximation.)
Somebody else could say “Wait, this ignore the Pauli exclusion principle” and the requirement that the electron wavefunction be fully antisymmetric. One could then make the (still single electron) model more complicated and construct a Slater determinant to use as a fully antisymmetric representation of the electron wavefunctions, generate the density, perform the self-consistent field computation to convergence. (This is Hartree-Fock.)
A third party could then note that this still underestimates what is called the “correlation energy” of the system, because treating the electron cloud as a continuous distribution through when electrons move ignores the fact that individual electrons strongly repel and hence do not like to get near one another. Both of the former approaches underestimate the size of the electron hole, and hence they make the atom “too small” and “too tightly bound”. A variety of schema are proposed to overcome this problem — using a semi-empirical local density functional being probably the most successful.
A fourth party might then observe that the Universe is really relativistic, and that by ignoring relativity theory and doing a classical computation we introduce an error into all of the above (although it might be included in the semi-empirical LDF approach heuristically).
In the end, one might well have an “ensemble” of models, all of which are based on physics. In fact, the differences are also based on physics — the physics omitted from one try to another, or the means used to approximate and try to include physics we cannot include in a first-principles computation (note how I sneaked a semi-empirical note in with the LDF, although one can derive some density functionals from first principles (e.g. Thomas-Fermi approximation), they usually don’t do particularly well because they aren’t valid across the full range of densities observed in actual atoms). Note well, doing the precise computation is not an option. We cannot solve the many body atomic state problem in quantum theory exactly any more than we can solve the many body problem exactly in classical theory or the set of open, nonlinear, coupled, damped, driven chaotic Navier-Stokes equations in a non-inertial reference frame that represent the climate system.
Note well that solving for the exact, fully correlated nonlinear many electron wavefunction of the humble carbon atom — or the far more complex Uranium atom — is trivially simple (in computational terms) compared to the climate problem. We can’t compute either one, but we can come a damn sight closer to consistently approximating the solution to the former compared to the latter.
So, should we take the mean of the ensemble of “physics based” models for the quantum electronic structure of atomic carbon and treat it as the best prediction of carbon’s quantum structure? Only if we are very stupid or insane or want to sell something. If you read what I said carefully (and you may not have — eyes tend to glaze over when one reviews a year or so of graduate quantum theory applied to electronics in a few paragraphs, even though I left out perturbation theory, Feynman diagrams, and ever so much more:-) you will note that I cheated — I run in a semi-empirical method.
Which of these is going to be the winner? LDF, of course. Why? Because the parameters are adjusted to give the best fit to the actual empirical spectrum of Carbon. All of the others are going to underestimate the correlation hole, and their errors will be systematically deviant from the correct spectrum. Their mean will be systematically deviant, and by weighting Hartree (the dumbest reasonable “physics based approach”) the same as LDF in the “ensemble” average, you guarantee that the error in this “mean” will be significant.
Suppose one did not know (as, at one time, we did not know) which of the models gave the best result. Suppose that nobody had actually measured the spectrum of Carbon, so its empirical quantum structure was unknown. Would the ensemble mean be reasonable then? Of course not. I presented the models in the way physics itself predicts improvement — adding back details that ought to be important that are omitted in Hartree. One cannot be certain that adding back these details will actually improve things, by the way, because it is always possible that the corrections are not monotonic (and eventually, at higher orders in perturbation theory, they most certainly are not!) Still, nobody would pretend that the average of a theory with an improved theory is “likely” to be better than the improved theory itself, because that would make no sense. Nor would anyone claim that diagrammatic perturbation theory results (for which there is a clear a priori derived justification) are necessarily going to beat semi-heuristic methods like LDF because in fact they often do not.
What one would do in the real world is measure the spectrum of Carbon, compare it to the predictions of the models, and then hand out the ribbons to the winners! Not the other way around. And since none of the winners is going to be exact — indeed, for decades and decades of work, none of the winners was even particularly close to observed/measured spectra in spite of using supercomputers (admittedly, supercomputers that were slower than your cell phone is today) to do the computations — one would then return to the drawing board and code entry console to try to do better.
Can we apply this sort of thoughtful reasoning the spaghetti snarl of GCMs and their highly divergent results? You bet we can! First of all, we could stop pretending that “ensemble” mean and variance have any meaning whatsoever by not computing them. Why compute a number that has no meaning? Second, we could take the actual climate record from some “epoch starting point” — one that does not matter in the long run, and we’ll have to continue the comparison for the long run because in any short run from any starting point noise of a variety of sorts will obscure systematic errors — and we can just compare reality to the models. We can then sort out the models by putting (say) all but the top five or so into a “failed” bin and stop including them in any sort of analysis or policy decisioning whatsoever unless or until they start to actually agree with reality.
Then real scientists might contemplate sitting down with those five winners and meditate upon what makes them winners — what makes them come out the closest to reality — and see if they could figure out ways of making them work even better. For example, if they are egregiously high and diverging from the empirical data, one might consider adding previously omitted physics, semi-empirical or heuristic corrections, or adjusting input parameters to improve the fit.
Then comes the hard part. Waiting. The climate is not as simple as a Carbon atom. The latter’s spectrum never changes, it is a fixed target. The former is never the same. Either one’s dynamical model is never the same and mirrors the variation of reality or one has to conclude that the problem is unsolved and the implementation of the physics is wrong, however “well-known” that physics is. So one has to wait and see if one’s model, adjusted and improved to better fit the past up to the present, actually has any predictive value.
Worst of all, one cannot easily use statistics to determine when or if one’s predictions are failing, because damn, climate is nonlinear, non-Markovian, chaotic, and is apparently influenced in nontrivial ways by a world-sized bucket of competing, occasionally cancelling, poorly understood factors. Soot. Aerosols. GHGs. Clouds. Ice. Decadal oscillations. Defects spun off from the chaotic process that cause global, persistent changes in atmospheric circulation on a local basis (e.g. blocking highs that sit out on the Atlantic for half a year) that have a huge impact on annual or monthly temperatures and rainfall and so on. Orbital factors. Solar factors. Changes in the composition of the troposphere, the stratosphere, the thermosphere. Volcanoes. Land use changes. Algae blooms.
And somewhere, that damn butterfly. Somebody needs to squash the damn thing, because trying to ensemble average a small sample from a chaotic system is so stupid that I cannot begin to describe it. Everything works just fine as long as you average over an interval short enough that you are bound to a given attractor, oscillating away, things look predictable and then — damn, you change attractors. Everything changes! All the precious parameters you empirically tuned to balance out this and that for the old attractor suddenly require new values to work.
This is why it is actually wrong-headed to acquiesce in the notion that any sort of p-value or Rsquared derived from an AR5 mean has any meaning. It gives up the high ground (even though one is using it for a good purpose, trying to argue that this “ensemble” fails elementary statistical tests. But statistical testing is a shaky enough theory as it is, open to data dredging and horrendous error alike, and that’s when it really is governed by underlying IID processes (see “Green Jelly Beans Cause Acne”). One cannot naively apply a criterion like rejection if p < 0.05, and all that means under the best of circumstances is that the current observations are improbable given the null hypothesis at 19 to 1. People win and lose bets at this level all the time. One time in 20, in fact. We make a lot of bets!
So I would recommend — modestly — that skeptics try very hard not to buy into this and redirect all such discussions to questions such as why the models are in such terrible disagreement with each other, even when applied to identical toy problems that are far simpler than the actual Earth, and why we aren’t using empirical evidence (as it accumulates) to reject failing models and concentrate on the ones that come closest to working, while also not using the models that are obviously not working in any sort of “average” claim for future warming. Maybe they could hire themselves a Bayesian or two and get them to recompute the AR curves, I dunno.
It would take me, in my comparative ignorance, around five minutes to throw out all but the best 10% of the GCMs (which are still diverging from the empirical data, but arguably are well within the expected fluctuation range on the DATA side), sort the remainder into top-half models that should probably be kept around and possibly improved, and bottom half models whose continued use I would defund as a waste of time. That wouldn’t make them actually disappear, of course, only mothball them. If the future climate ever magically popped back up to agree with them, it is a matter of a few seconds to retrieve them from the archives and put them back into use.
Of course if one does this, the GCM predicted climate sensitivity plunges from the totally statistically fraudulent 2.5 C/century to a far more plausible and still possibly wrong ~1 C/century, which — surprise — more or less continues the post-LIA warming trend with a small possible anthropogenic contribution. This large a change would bring out pitchforks and torches as people realize just how badly they’ve been used by a small group of scientists and politicians, how much they are the victims of indefensible abuse of statistics to average in the terrible with the merely poor as if they are all equally likely to be true with randomly distributed differences.
Reader Comments (89)
I also follow Brown's comments.
Antony Watts posted a well written essay by Brown on Paul Bain’s use of ‘denier’ in the scientific literature http://wattsupwiththat.com/2012/06/22/a-response-to-dr-paul-bains-use-of-denier-in-scientific-literature/
Following this Brown had an exchange of views at 'Climate Abyss' with John Nielsen-Gammon: Skeptics Are Not Deniers: A Conversation in 4 parts. http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-1/
Well worth a read for those who haven't seen it.
Al.
"The problem with calibrating anything against the actual climate record is that no such thing exists in any meaningful way. "
That is exactly why predicting future temperatures is a silly fantasy. Something like 70% of the earth is water and there is no way to know what temperatures above it were in the past. So 70% of the earth's temperature is unknown before satellites. Roughly. Totally unknown in pre measurement times.
There is no consensus between the model outputs.
97% of all the models disagree with the rest of them.
There was actually 6 parts:
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-1/
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-2/
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-3/
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-4/
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-5/
http://blog.chron.com/climateabyss/2012/07/skeptics-are-not-deniers-a-conversation-part-6/
It seems that climatology has taken its basic method from the phenomenon known as the "wisdom of crowds." You know, when a crowd makes guesses of the number of jelly beans in a jar or the weight of a cow, the average comes quite close to the truth. Of course, that is an admission that the models are only guesses, and we have "only" 73 of them.
And don't forget the "black box" analysis of GCMs by Willis. It showed that at bottom, all of them are making the same guess: that temperatures will rise in concert with CO2 concentrations.
@ Richard Verney, 12:21
"When you have say for example 73 models (this was the subject of a discussion by Dr Roy Spencer some months back), and they all output a different result, you know, as fact, that 72 of the 73 models must be wrong."
It's even worse than that!
How can you know as a fact that even one model is correct?
"How can you know as a fact that even one model is correct?"
.......well I suppose it has to get the average temperature of the world about right, which narrows the field down quite considerably: (or did in 2009):
http://rankexploits.com/musings/2009/fact-6a-model-simulations-dont-match-average-surface-temperature-of-the-earth/
John Marshall
Excellent. You can tell me. Where does all that energy not leaving in the 13-17 micrometre band of the OLR actually go?
That damned butterfly http://www.mediafire.com/view/u5ar9ucd8n11fma/Damned_Butterfly.JPG
Bish
very glad you posted this as Robert Brown is undoubtedly one of the greatest assets of the 'well-funded and highly co-ordinated' sceptical movement - when he is not 'grading' papers. Dont you just love American English!
My only quibble with this post is that he thinks it worthwhile retaining any of the wretched models. They should all be ditched and the modellers should get real jobs - adding to the wealth of the world rather than destroying it.
Sorry Tamsin you had a chance to tell us the many and varied tangible benefits the billions spent on modelling have brought us but I have interpreted your silence as evidence of a complete lack of such benefits.
rgb writes: ". . . it looks like the frayed end of a rope, not like a coherent spread around some physics supported result."
I love that, "the frayed end of a rope." Rarely do you see such a descriptive phrase in the debate on climate.
pottereaton
You'd need to bitch-slap a sufficiently large number and take the mean in order to be significant.
Jun 14, 2013 at 1:13 PM | Unregistered Commenter rhoda
I fell off my ladder when I read that one. Brilliant. Rhoda
rgb@duke is one of the commenters I specifically look out for. His posts are extremely well written and extremely well argued. What a pitty his writings are not more widely disseminated.
This is exactly the kind of professor I wish I had come into contact with at university, trully inspiring.
rgb@duke had some superb posts in the skydragon fights over on WUWT in the not-too-distant past. He clearly shines as a teacher, but unfortunately the stubborn stupidity of the contrarians could not be defeated.
Apple has just announced the newest iteration of their 'Mac Pro' workstation. If it had existed in 2003 (only ten years ago) it would have ranked as the 8th fastest supercomputer in the world! Perhaps it's not totally relevant, but it does give an idea of how science can progress when it's given the chance...
I agree Fred - not relevant. Comes across as product placement.
Fred Jensen re Mac Pro
Using these useless Intel processors that Steve Jobs said were 3 times slower than Motorola. There are liars everywhere and most of them have a fanbase.
There is a vast monoculture out there whose members use a very high percentage of their brain to process a single word - 'cool'. Being cool is being superior, not one of the dumb masses. See Nietzsche, deep ecology and Apple's entire raison d'etre for details.
Each of those contains somewhat different physics. The basic physical laws that the models try to capture are the same, but the way the models try to incorporate those laws differs. This is because the climate system is hugely more complicated than the largest computers can capture, so some phenomena have to be parameterized. These result in the different computed climatic reactions to the “forcing” of the climate by humanity and the natural drivers that are incorporated in the forecasts. When you look at the graph it is clear that some models do quite a bit better than the bulk of the ensemble. In fact, several models cannot be distinguished from reality by the usual statistical criteria.
You know that what is done in science is to throw out the models that don’t fit the data and keep for now the ones that seem to be consistent with the observational data. In this way you can learn why it is that some models do better than others and make progress in understanding the dynamics of the system. This is research.
But this is not what is done. What you read in IPCC reports is that all the models are lumped into a statistical ensemble, as if each model were trying to “measure” the same thing, and all variations are a kind of noise, not different physics. This generates the solid black line as a mean of the ensemble and a large envelope of uncertainty. The climate sensitivity and its range of uncertainty contained in the reports are obtained in this way. This enables all of the models to keep their status. But ultimately as the trend continues, it becomes obvious that the ensemble and its envelope are emerging out of the observational “signal.” This is where we are now.
Noblesse Oblige - in your first paragraph, are you speaking from a position of informed knowledge or are you speculating? Did you see Roy Spencer's post on tropospheric trends versus radiosonde data?
http://www.drroyspencer.com/2013/06/epic-fail-73-climate-models-vs-observations-for-tropical-tropospheric-temperature/
RgbatDuke is the model of what a scientist is supposed to be. His independent thought is the model of how AGW/CAGW should be discussed and debated.
Noblesse Oblige
Is "parameterized" a nice shiny sciency word for "guessed at", do you think?
Look at every estimated parameter and question the validity of their estimates.
Jun 14, 2013 at 10:18 AM | Unregistered CommenterPeter Stroud
Are you sure you want this? It would keep them all employed past retirement.
Jun 14, 2013 at 12:10 PM | Rick Bradford
Elbonian sprang to mind when I saw Huppert's picture:
http://dilbert.com/strips/comic/2008-10-03/
A couple of often overlooked issues with climate models is that they actually produce a significantly wider range of absolute temperatures than they do anomalies (and they produce hindcasts of quite different planets in terms of absolute temperatures), and they do not incorporate uncertainty explicitly into their various sub-models and bring this through.
A couple of often overlooked issues with climate models is that they actually produce a significantly wider range of absolute temperatures than they do anomalies (and they produce hindcasts of quite different planets in terms of absolute temperatures), and they do not incorporate uncertainty explicitly into their various sub-models and bring this through.
Jun 14, 2013 at 5:12 PM | Noblesse Oblige
I believe that Gavin Schmidt has posted that the ensemble approach is good because the spaghetti shows the noise while the average reveals the warming signal. Upon first reading, I knew that this claim had no foundation in statistics, scientific method, or science. Here is Schmidt quoted by Bob Tisdale:
"The following quotes are from the thread of the RealClimate post Decadal predictions. At comment 49, dated 30 Sep 2009 at 6:18 AM, a blogger posed this question:
If a single simulation is not a good predictor of reality how can the average of many simulations, each of which is a poor predictor of reality, be a better predictor, or indeed claim to have any residual of reality?
Gavin Schmidt replied:
Any single realisation can be thought of as being made up of two components – a forced signal and a random realisation of the internal variability (‘noise’). By definition the random component will uncorrelated across different realisations and when you average together many examples you get the forced component (i.e. the ensemble mean)."
http://bobtisdale.wordpress.com/2013/01/09/satellite-era-model-data-precipitation-comparison-for-the-uk-and-us/
Jun 14, 2013 at 3:53 PM | ntropywins
I too love American English. But it is fast disappearing into the maw of Political Correctness. Enjoy it while you can. Today, I enjoy British English far more than American English. The Brits continue to permit a bit of seasoning. Delingpole makes me smile. There are no Delingpoles in the US today.
I fear that rgbatduke will be hauled before Duke's "Diversity Dean" for having used the phrase "bitch-slap."
Or, to paraphrase, you will always see the effect which they programmed in. Especially if you filter the results for 'outliers' which you pick by their failure to reproduce warming.
"Everything works just fine as long as you average over an interval short enough that you are bound to a given attractor, oscillating away, things look predictable and then — damn, you change attractors. Everything changes! All the precious parameters you empirically tuned to balance out this and that for the old attractor suddenly require new values to work."
YES! Exactly! Which is also why we see all those various "step changes" that people wonder about. What would be surprising would be if we did not see such steps.
@Jun 14, 2013 at 2:46 PM | Joe Public
/////////////////////
Joe
That was the very point that I was making in my 3rd paragraph.(to which I should probably have added, the admitted inability to properly model clouds).
Gavin Schmidt quoted as saying: "Any single realisation can be thought of as being made up of two components – a forced signal and a random realisation of the internal variability (‘noise’). By definition the random component will uncorrelated across different realisations and when you average together many examples you get the forced component (i.e. the ensemble mean)."
Which means that because the ensemble mean comprising the forced signal is clearly diverging from reality as shown by measured temperature observations of the last 20 years, it can only be concluded that the forced component on the models must be wrong. As they are all supposed to be based on the same "known" physics it is hard not to conclude that the physics must be wrong.
The forced signal is prescribed, that's what make the climate models useless.
Models are getting an awful reputation, mainly thanks to "prominent" climate modellers, and there is where I resent most the silence of other climate modelers. The reason is that models are very useful tools, but their utility lies in showing us where sre we wrong. The moment the model stop doing that and becomes an oracle, you can be pretty sure you are not doing science any more.
The very accurate mean of a crap result is nothing but crap with a lot of decimals, and it is a very old fashion attitude that the populace is going to accept it because it has so many digits after the decimal point, so let's get real. It is not a problem with the parameterizations, some of them are quite poor, but some others are brilliant. The problem is that we (some we) attribute more precission to models than to the most precise instruments we have. That does not make sense.
Even if the CO2 hypothesis as ruler of climate were remotely tenable, models cannot proof it.
The impact of this secondary trace gas is less than 1% of the external radiative forcing of this planet, the WMO certified First Class instruments have less accuracy, and the real instruments deployed out there and measuring data can have 10 times less accuracy. How can we believe that a mathematical abstraction based on poor accuracy is going to be precise?
The only reasonable answer that I can find is faith, which is far more common among scientists than one would like to think.
This has legs.
http://wattsupwiththat.com/2013/06/16/quote-of-the-week-a-pause-for-cooling-off/
Robert Brown is one of my favorite commentors. He's one guy that could get me back into the classroom for a physics course.
Robert Brown is one of my favorite commentors. He's one guy that could get me back into the classroom for a physics course.
Is "parameterized" a nice shiny sciency word for "guessed at", do you think?
Jun 14, 2013 at 7:26 PM Mike Jackson
No. It means "grossly oversimplified".
"'ibbert's a bleedin' ginger 'e is. Yer carn't take nuffin' as gorspell from gingers, coz they is barmy as bedbugs."
Plus, we ain't got no soul, says my teenage daughter.