Bishop Hill On the meaning of ensemble means
Jun 14, 2013 Climate: Models Climate: Statistics Readers have been pointing me to this comment at WUWT. It's written by a reader calling themselves rgbatduke, and it considers the mean and standard deviation of an ensemble of GCM predictions and asks whether these figures have any meaning. It concludes that they do not.
Saying that we need to wait for a certain interval in order to conclude that “the models are wrong” is dangerous and incorrect for two reasons. First — and this is a point that is stunningly ignored — there are a lot of different models out there, all supposedly built on top of physics, and yet no two of them give anywhere near the same results!
This is reflected in the graphs Monckton publishes above, where the AR5 trend line is the average over all of these models and in spite of the number of contributors the variance of the models is huge. It is also clearly evident if one publishes a “spaghetti graph” of the individual model projections (as Roy Spencer recently did in another thread) — it looks like the frayed end of a rope, not like a coherent spread around some physics supported result.
Note the implicit swindle in this graph — by forming a mean and standard deviation over model projections and then using the mean as a “most likely” projection and the variance as representative of the range of the error, one is treating the differences between the models as if they are uncorrelated random variates causing >deviation around a true mean!.
Say what?
This is such a horrendous abuse of statistics that it is difficult to know how to begin to address it. One simply wishes to bitch-slap whoever it was that assembled the graph and ensure that they never work or publish in the field of science or statistics ever again. One cannot generate an ensemble of independent and identically distributed models that have different code. One might, possibly, generate a single model that generates an ensemble of predictions by using uniform deviates (random numbers) to seed
“noise” (representing uncertainty) in the inputs.
What I’m trying to say is that the variance and mean of the “ensemble” of models is completely meaningless, statistically because the inputs do not possess the most basic properties required for a meaningful interpretation. They are not independent, their differences are not based on a random distribution of errors, there is no reason whatsoever to believe that the errors or differences are unbiased (given that the only way humans can generate unbiased anything is through the use of e.g. dice or other objectively random instruments).
So why buy into this nonsense by doing linear fits to a function — global temperature — that has never in its entire history been linear, although of course it has always been approximately smooth so one can always do a Taylor series expansion in some sufficiently small interval and get a linear term that — by the nature of Taylor series fits to nonlinear functions — is guaranteed to fail if extrapolated as higher order nonlinear terms kick in and ultimately dominate? Why even pay lip service to the notion that 
Let me repeat this. It has no meaning! It is indefensible within the theory and practice of statistical analysis. You might as well use a ouija board as the basis of claims about the future climate history as the ensemble average of different computational physical models that do not differ by truly random variations and are subject to all sorts of omitted variable, selected variable, implementation, and initialization bias. The board might give you the right answer, might not, but good luck justifying the answer it gives on some sort of rational basis.
Let’s invert this process and actually apply statistical analysis to the distribution of model results Re: the claim that they all correctly implement well-known physics. For example, if I attempt to do an a priori computation of the quantum structure of, say, a carbon atom, I might begin by solving a single electron model, treating the electron-electron interaction using the probability distribution from the single electron model to generate a spherically symmetric “density” of electrons around the nucleus, and then performing a self-consistent field theory iteration (resolving the single electron model for the new potential) until it converges. (This is known as the Hartree approximation.)
Somebody else could say “Wait, this ignore the Pauli exclusion principle” and the requirement that the electron wavefunction be fully antisymmetric. One could then make the (still single electron) model more complicated and construct a Slater determinant to use as a fully antisymmetric representation of the electron wavefunctions, generate the density, perform the self-consistent field computation to convergence. (This is Hartree-Fock.)
A third party could then note that this still underestimates what is called the “correlation energy” of the system, because treating the electron cloud as a continuous distribution through when electrons move ignores the fact that individual electrons strongly repel and hence do not like to get near one another. Both of the former approaches underestimate the size of the electron hole, and hence they make the atom “too small” and “too tightly bound”. A variety of schema are proposed to overcome this problem — using a semi-empirical local density functional being probably the most successful.
A fourth party might then observe that the Universe is really relativistic, and that by ignoring relativity theory and doing a classical computation we introduce an error into all of the above (although it might be included in the semi-empirical LDF approach heuristically).
In the end, one might well have an “ensemble” of models, all of which are based on physics. In fact, the differences are also based on physics — the physics omitted from one try to another, or the means used to approximate and try to include physics we cannot include in a first-principles computation (note how I sneaked a semi-empirical note in with the LDF, although one can derive some density functionals from first principles (e.g. Thomas-Fermi approximation), they usually don’t do particularly well because they aren’t valid across the full range of densities observed in actual atoms). Note well, doing the precise computation is not an option. We cannot solve the many body atomic state problem in quantum theory exactly any more than we can solve the many body problem exactly in classical theory or the set of open, nonlinear, coupled, damped, driven chaotic Navier-Stokes equations in a non-inertial reference frame that represent the climate system.
Note well that solving for the exact, fully correlated nonlinear many electron wavefunction of the humble carbon atom — or the far more complex Uranium atom — is trivially simple (in computational terms) compared to the climate problem. We can’t compute either one, but we can come a damn sight closer to consistently approximating the solution to the former compared to the latter.
So, should we take the mean of the ensemble of “physics based” models for the quantum electronic structure of atomic carbon and treat it as the best prediction of carbon’s quantum structure? Only if we are very stupid or insane or want to sell something. If you read what I said carefully (and you may not have — eyes tend to glaze over when one reviews a year or so of graduate quantum theory applied to electronics in a few paragraphs, even though I left out perturbation theory, Feynman diagrams, and ever so much more:-) you will note that I cheated — I run in a semi-empirical method.
Which of these is going to be the winner? LDF, of course. Why? Because the parameters are adjusted to give the best fit to the actual empirical spectrum of Carbon. All of the others are going to underestimate the correlation hole, and their errors will be systematically deviant from the correct spectrum. Their mean will be systematically deviant, and by weighting Hartree (the dumbest reasonable “physics based approach”) the same as LDF in the “ensemble” average, you guarantee that the error in this “mean” will be significant.
Suppose one did not know (as, at one time, we did not know) which of the models gave the best result. Suppose that nobody had actually measured the spectrum of Carbon, so its empirical quantum structure was unknown. Would the ensemble mean be reasonable then? Of course not. I presented the models in the way physics itself predicts improvement — adding back details that ought to be important that are omitted in Hartree. One cannot be certain that adding back these details will actually improve things, by the way, because it is always possible that the corrections are not monotonic (and eventually, at higher orders in perturbation theory, they most certainly are not!) Still, nobody would pretend that the average of a theory with an improved theory is “likely” to be better than the improved theory itself, because that would make no sense. Nor would anyone claim that diagrammatic perturbation theory results (for which there is a clear a priori derived justification) are necessarily going to beat semi-heuristic methods like LDF because in fact they often do not.
What one would do in the real world is measure the spectrum of Carbon, compare it to the predictions of the models, and then hand out the ribbons to the winners! Not the other way around. And since none of the winners is going to be exact — indeed, for decades and decades of work, none of the winners was even particularly close to observed/measured spectra in spite of using supercomputers (admittedly, supercomputers that were slower than your cell phone is today) to do the computations — one would then return to the drawing board and code entry console to try to do better.
Can we apply this sort of thoughtful reasoning the spaghetti snarl of GCMs and their highly divergent results? You bet we can! First of all, we could stop pretending that “ensemble” mean and variance have any meaning whatsoever by not computing them. Why compute a number that has no meaning? Second, we could take the actual climate record from some “epoch starting point” — one that does not matter in the long run, and we’ll have to continue the comparison for the long run because in any short run from any starting point noise of a variety of sorts will obscure systematic errors — and we can just compare reality to the models. We can then sort out the models by putting (say) all but the top five or so into a “failed” bin and stop including them in any sort of analysis or policy decisioning whatsoever unless or until they start to actually agree with reality.
Then real scientists might contemplate sitting down with those five winners and meditate upon what makes them winners — what makes them come out the closest to reality — and see if they could figure out ways of making them work even better. For example, if they are egregiously high and diverging from the empirical data, one might consider adding previously omitted physics, semi-empirical or heuristic corrections, or adjusting input parameters to improve the fit.
Then comes the hard part. Waiting. The climate is not as simple as a Carbon atom. The latter’s spectrum never changes, it is a fixed target. The former is never the same. Either one’s dynamical model is never the same and mirrors the variation of reality or one has to conclude that the problem is unsolved and the implementation of the physics is wrong, however “well-known” that physics is. So one has to wait and see if one’s model, adjusted and improved to better fit the past up to the present, actually has any predictive value.
Worst of all, one cannot easily use statistics to determine when or if one’s predictions are failing, because damn, climate is nonlinear, non-Markovian, chaotic, and is apparently influenced in nontrivial ways by a world-sized bucket of competing, occasionally cancelling, poorly understood factors. Soot. Aerosols. GHGs. Clouds. Ice. Decadal oscillations. Defects spun off from the chaotic process that cause global, persistent changes in atmospheric circulation on a local basis (e.g. blocking highs that sit out on the Atlantic for half a year) that have a huge impact on annual or monthly temperatures and rainfall and so on. Orbital factors. Solar factors. Changes in the composition of the troposphere, the stratosphere, the thermosphere. Volcanoes. Land use changes. Algae blooms.
And somewhere, that damn butterfly. Somebody needs to squash the damn thing, because trying to ensemble average a small sample from a chaotic system is so stupid that I cannot begin to describe it. Everything works just fine as long as you average over an interval short enough that you are bound to a given attractor, oscillating away, things look predictable and then — damn, you change attractors. Everything changes! All the precious parameters you empirically tuned to balance out this and that for the old attractor suddenly require new values to work.
This is why it is actually wrong-headed to acquiesce in the notion that any sort of p-value or Rsquared derived from an AR5 mean has any meaning. It gives up the high ground (even though one is using it for a good purpose, trying to argue that this “ensemble” fails elementary statistical tests. But statistical testing is a shaky enough theory as it is, open to data dredging and horrendous error alike, and that’s when it really is governed by underlying IID processes (see “Green Jelly Beans Cause Acne”). One cannot naively apply a criterion like rejection if p < 0.05, and all that means under the best of circumstances is that the current observations are improbable given the null hypothesis at 19 to 1. People win and lose bets at this level all the time. One time in 20, in fact. We make a lot of bets!
So I would recommend — modestly — that skeptics try very hard not to buy into this and redirect all such discussions to questions such as why the models are in such terrible disagreement with each other, even when applied to identical toy problems that are far simpler than the actual Earth, and why we aren’t using empirical evidence (as it accumulates) to reject failing models and concentrate on the ones that come closest to working, while also not using the models that are obviously not working in any sort of “average” claim for future warming. Maybe they could hire themselves a Bayesian or two and get them to recompute the AR curves, I dunno.
It would take me, in my comparative ignorance, around five minutes to throw out all but the best 10% of the GCMs (which are still diverging from the empirical data, but arguably are well within the expected fluctuation range on the DATA side), sort the remainder into top-half models that should probably be kept around and possibly improved, and bottom half models whose continued use I would defund as a waste of time. That wouldn’t make them actually disappear, of course, only mothball them. If the future climate ever magically popped back up to agree with them, it is a matter of a few seconds to retrieve them from the archives and put them back into use.
Of course if one does this, the GCM predicted climate sensitivity plunges from the totally statistically fraudulent 2.5 C/century to a far more plausible and still possibly wrong ~1 C/century, which — surprise — more or less continues the post-LIA warming trend with a small possible anthropogenic contribution. This large a change would bring out pitchforks and torches as people realize just how badly they’ve been used by a small group of scientists and politicians, how much they are the victims of indefensible abuse of statistics to average in the terrible with the merely poor as if they are all equally likely to be true with randomly distributed differences.
Reader Comments (89)
Some of those models are only in the ensemble based on the prestige of the national institutions which support them, not on any real merit. .Chuck them out, find the best few, give us your best shot.
rgbatduke would be this guy.
...First of all, we could stop pretending that “ensemble” mean and variance have any meaning whatsoever by not computing them. Why compute a number that has no meaning? ...
Er... I don't think that you understand Climate Science very well!
You appear to be doing real science. This is not acceptable in the climate world. It will NOT produce the required output, which is a grant continuation.
To answer your question above: "Why compute these numbers?" - the answer is because the result is a SINGLE number, and so can be constrained using semi-random statistical manipulations to become any figure that is desired. And the ability to produce a specific number on demand is much prized by the IPCC, who depend for their very existence on the continuous production of papers which have a desired set of numbers in them.
I trust that this answers your question. You may be a capable statistician, but you are woefully ignorant about the business of making money in the real world of science...
/sarc
Yes, this is right is it not? The very simple proposal is that if we are looking for models which approximate a phenomenon, the way to go is rate the various possibilities in order of how well they actually approximate, and throw out the worse ones. Other than for purposes of discovering what their source of error is.
It always has struck one as utterly weird, this process that IPCC and others seem to go through, averaging a lot of more or less decent models with ones that don't seem to be working at all. It would never be allowed for any decision making purpose in drug trials. Its actually a license to manipulate the ensemble mean however one wants and in defiance of actual data.
Think about it. If your predictions can be generated by averaging good and bad models, how do you pick the bad ones? They can have anything at all in them. Its crazy.
And this is why we are covering the country with wind turbines? Completely mad. We are using admittedly bad models, and even according to them, it will have no effect on warming? So we cover the country with turbines and raise electricity prices... for what?
The follow-up comments from RGB are perhaps more easily comprehensible:
http://wattsupwiththat.com/2013/06/13/no-significant-warming-for-17-years-4-months/#comment-1335179
http://wattsupwiththat.com/2013/06/13/no-significant-warming-for-17-years-4-months/#comment-1335200
"Does climate science truly stand alone in failing to recognize unrealistic behavior when it bites it in the ass? "
"the community needs to work a bit harder and faster to fix this in AR5 and needs to swallow their pride and be the ones to announce to the media that perhaps the “catastrophe” they predicted ten years ago was a wee bit exaggerated. "
http://wattsupwiththat.com/2013/06/13/no-significant-warming-for-17-years-4-months/#comment-1335214
"It’s high time that it was pointed out that this average is a completely meaningless quantity, and that 2/3 of the spaghetti needs to go straight into the toilet as failed, not worth the energy spent running the code. But if they did that, 2.5 C would “instantly” turn into 1-1.5C, or even less, and this would be the equivalent of Mount Tambora exploding under the asses of climate scientists everywhere, an oops so big that nobody would ever trust them again."
"And somewhere, that damn butterfly."
I'm going to frame this and hang it on my wall.
A good, rational, scientific analysis, although - or rather, hence - probably irrelevant to the climate argument. (Another post at WUWT apparently featuring a delegate - admittedly a very young and foolish delegate - giving her views on how the German summer has been cooled by global warming shows what one's up against)
Nonetheless, the writer would have been welcome at Cheltenham for last week's rather anaemic discussion of climate models.
I would be very interested to hear what Tamsin (and Richard) thinks about this comment.
I have a considerable amount of second hand experience of modelling environmental fluid dynamics, having been responsible for a research team in this area for a number of years. The analysis by rgbatduke chimes with my own view of model utilisation but seems a million miles away from current practice in Climate Science
Related reading:
Reliability of the CMIP3 ensemble, by Annan and Hargreaves
Second-order exchangeability analysis for multi-model ensembles, by Rougier et al.
Robert G. Brown's Home Page - Duke Physics
https://www.google.com.au/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CC4QFjAA&url=http%3A%2F%2Fwww.phy.duke.edu%2F~rgb%2F&ei=Jdy6Ub6zAYPpiAfF7oHABg&usg=AFQjCNGhcCxIPMtMGXlvM3ovPn6t8tsaMQ&sig2=UFOQWDGoLoKALRNptUX1rg&bvm=bv.47883778,d.aGc
The man's a hero !
The post points out that the calculated variance of the simulated temperature series has no meaning. The reason is that, in order to calculate the variance of the series, we first have to have a statistical model of the data—and the statistical model that has been used is garbage.
That is very similar to the main point in the recent discussion about the significance of an increase in the observed temperature series. The general point is this: in order to draw statistical inferences (e.g. what the variance is, or whether an increase is significant), we have to first select a statistical model of the data—and the statistical models that have been used in climate science (e.g. for measured temperatures, for simulated temperatures) are unjustifiable. Hence all the statistical inferences are ill-founded.
This fundamental error occurs throughout climate science. And it occurs in the work of skeptics as well as alarmists. If you see some analysis that claims to have found something about a variance, or significance, or “unit root”, etc., the analysis is wrong, because it is based on wrong assumptions, i.e. an unjustifiable statistical model. If the statistical model is unjustified, as it virtually always is, then the statistical inferences are unfounded. Hardly anyone, on either side of the debate, wants to face this.
The truth contained in this items should be evident to any real scientist who reads it with scientific objectivity.
I follow comments by rgbatduke on WUWT and find them very pertinent and with a good sense of humour.
I too think it's Robert Brown, based on his previous postings and his background.
Aerthur Dent: I agree with you, having also worked in modelling of fluid dynamics for many years and having experience of many different computer models.
But, as has been pointed outmany times, none of the models are able to predict anything like the real world empirical data. None were able to forecast the current hiatus in global mean surface temperatures. So what is the point of further analysis?
Surely it is up to the modellers to examine the parameters that they were forced to estimate/guess. Then perhaps think, what is to them the unthinkable: feedback might not be as large: might not even be positive. Look at every estimated parameter and question the validity of their estimates.
Oh, and whilst they are reconsidering their science, suggest to the policy makers that things night well be less worrying than first thought.
I'm sure a climate psientist will be able to explain why rgbatduke is totally wrong.
Here are some possibilities;
(S)He is not a climate psientist
Only climate psientists can understand the complexity of the psience
(S)He is a denier.
Did I miss anything?
OT slightly but news of my local MP, Dr. Julian Huppert.
http://www.cambridge-news.co.uk/Cambridge/Cambridge-MP-Julian-Huppert-accuses-groaning-rivals-of-bullying-20130613093625.htm
This is a guy who I have posted about in the past about his deep green, or should I say "groan" views?
Unfortunately in Cambridge such people are far too common. He fits right in.
Yes - it is R G Brown - a physicist at Duke University. I've read a lot of his posts and he is a very smart cookie.
There's been a lot of talk about science by consensus at various times, and discussed at length how science is not progressed by consensus.
Taking a mean and variance of of the 'ensemble' looks scientific, but all it is, is a not so subtle form of consensus. Thanks rgbatduke for pointing out how wrong it is. A great article. And thanks Andrew for posting it.
rgbatduke would be this guy.
Jun 14, 2013 at 9:33 AM | Registered CommenterPaul Matthews
Confirmed
Mr Bat-Duke is dead right.
The contortions we see from the 'models' makes as much sense as averaging all your lottery predictions and then betting on 17,18,19,20,21,22,23.
Then acting surprised when they don't come up.
RGB is Dr Robert Brown at Duke Uni. One of the very few physicists on the blogs that really understands his subject and can communicate it clearly.
Hi,
thanks for asking Arthur.
I've only read the first bit of the post I'm afraid - various tasks on the to-do list for this week (hence quiet generally). Just a couple of quick comments on the non-independence of models.
(a) climate scientists are well aware the models are not independent (e.g. papers by Reto Knutti), but
(b) don't agree on the best way to estimate their correlations or (related) the degree to which the ensemble variance represents our actual uncertainty about the real world (this is a problem!!), and
(c) Jonty et al. have made a new stab at this problem, which Bish cites (Second order exchangeability...) and I'm working with him to continue this line of work.
Tamsin? Anything to add?
The most relevant person to comment might be Ed Hawkins.
He is a contributing author to the IPCC AR5 chapter 11 that is being criticised.
Your Grace
Off Topic, but I consider that you should have a post on this forthcoming meeting.
http://www.dailymail.co.uk/news/article-2341484/Floods-droughts-snow-May-Britains-weather-got-bad-Met-Office-worried.html
There is quite a bit to discuss so it would make an interesting post.
Once people's views/comments have been assembled, someone should write to the Met Office to steer them onto the right track by suggesting that they concentrate on CET to see the extent of variability in UK climate, its trends and over lay this with CO2 emissions (there being no correlation between CO2 levels and the rise of CET from its earliest records).
Sorry to add to the happy clappy club, but I have an rb@duke text file, with the collected posts.
I not that the bio posted by Paul Matthews (above) has:
Says it all, really.
I have asked many times, in my own inarticulate way, who audits the models, when are they assess for relevance and fitness. You have here spelled it out in a fine manner, what I have been trying to say.
There needs to be a process to weed out the inappropriate models but is the IPCC, and academia, up to such necessary auditing?
So you mean ''garbage in: garbage out''.
Climate sensitivity is calculated assuming CO2 drives climate. It does not. None of the empirical data supports such a claim.
Many thanks for your hard work and now sore fingers from the keyboard bashing.
Somebody please send Private Eye a lookalike between Huppert and Fungus the Bogeyman.
http://en.wikipedia.org/wiki/Fungus_the_Bogeyman
http://www.cambridge-news.co.uk/Cambridge/Cambridge-MP-Julian-Huppert-accuses-groaning-rivals-of-bullying-20130613093625.htm
Although the subject is a little over my head his explanation is clear and illuminating.
What a shame that a field that has produced such great work can be so abused.
This on survivor bias is an interesting read:
http://youarenotsosmart.com/2013/05/23/survivorship-bias/
In New York City, in an apartment a few streets away from the center of Harlem, above trees reaching out over sidewalks and dogs pulling at leashes and conversations cut short to avoid parking tickets, a group of professional thinkers once gathered and completed equations that would both snuff and spare several hundred thousand human lives.
This para is probably more catchy:
The official name for the people inside the apartment was the Statistical Research Group, a cabal of geniuses assembled at the request of the White House and made up of people who would go on to compete for and win Nobel Prizes. The SRG was an extension of Columbia University, and they dealt mainly with statistical analysis. The Philadelphia Computing Section, made up entirely of women mathematicians, worked six days a week at the University of Pennsylvania on ballistics tables. Other groups with different specialties were tied to Harvard, Princeton, Brown and others, 11 in all, each a leaf at the end of a new branch of the government created to help defeat the Axis – the Department of War Math.
@tckev
'There needs to be a process to weed out the inappropriate models but is the IPCC, and academia, up to such necessary auditing?'
In the real world, you would ask who is independent enough to take on this task? And it sure as heck isn't going to be climate modellers themselves. They have too much to lose in terms of career opportunities. Same reason as solicitors never openly criticise other lawyers...they may have t deal with them on a different basis in the future.
And on a slightly less cynical note, the current reward structure fro academic scientist doesn't reward them for doing so either. Why should they spend time on finding the holes in Bloggs' model whne that could be better spent on their own and maybe bring something they do get credit for nearer to fruition.
It is not difficult stuff to comprehend, but people do what you 'pay' them to do (even if the 'pay' isn't in actual cash). If you 'pay' them to publish papers, they'll publish papers. If, instead you paid them to criticise others then that's what they'd do.
But at the moment publication and citation are the big ticket items. Not 'auditing'.
"....One simply wishes to bitch-slap whoever it was that assembled the graph and ensure that they never work or publish in the field of science or statistics ever again...."
Time to find out 'whoever it was' so that they can 'Duke' it out.
My money's on the Duke!
On a number of occassions when commenting upon models, I have said that they remind me of the Dire Strait lyric "Two men say they're Jesus, one of them must be wrong".
When you have say for example 73 models (this was the subject of a discussion by Dr Roy Spencer some months back), and they all output a different result, you know, as fact, that 72 of the 73 models must be wrong.
Does that inspire confidence that one of the 73 models is right? Clearly, it does not, since the most obvious reson why 72 of the 73 models is wrong is that the basic physics underpinning the models is simply wrong, and/or there is an inability to deal properly with the water cycle and the complexities of the ocean, and/or due to chaos prediction (projection) is simply impossible. The simple fact taht all models disagree with one another, strongly suggests that they are all wrong; there should only be modest differences pertaining to assumptions made to the use of different forcings 9eg climate sensitivity to CO2, aerosol forcing etc).
As I repeatedly say when discussing models, cliamte science will only advance once the models are ditched. Presently they are useless.
Most studies are based not upon the collection and study of empirical data/observation, but rather upon model projectsions. Since we know that all the models are wrong, what is the point of a study based upon models? Models wrong, equals study wrong!
Averaging is the other bug bear in climate science. It is vitakl to get away from averaging since this hides what is going on; the one thing about the average condition is that it is rarely encountered.
What is the average temperature, rainfall, solar insolation, DWLWIR, concentration and distribution of CO2, albedo, humidity etc and where on planet Earth is that average found?
Rick
That position's already taken by Charles Clarke (don't click if you're of a nervous disposition).
However, FYI, Lord Gnome's address is strobes@private-eye.co.uk
Indeed, analysis of model ensembles is in essence sociology of the climate modelling community; it has little relationship to physical science.
I've always wondered why obviously poorly performing models were not weeded out. The only reason I can think of is that it would be too damaging to consensus-building between the different groups. Something like an IPCC process would not be impossible.
One possible reason for averaging over the models would be the wisdom of crowds idea: (From Wikipedia https://en.wikipedia.org/wiki/The_Wisdom_of_Crowds) : "the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox's true butchered weight than the estimates of most crowd members, and also closer than any of the separate estimates made by cattle experts)."
This does not justify doing a full statistical analysis on the "ensemble" of models' outputs, but perhaps there might be some use for the average. Then again, Surowski reckons that each crowd member's opinion must not be determined by the opinions of those around them, and I'm not sure how independent the models are at this time
Ah, it's just clicked into place I understand now:
97% of climate
scientistsmodelsagreedisagree....Many a time I have seen an rgb comment at wuwt that has been worthy of a post in it's own right. Anthony has in fact elevated many such comments. Lectures at Duke must be fun. I'm sure glad that he's not on the warmist's side.
That you cannot apply frequentist stats to such models was already standardised in the peer reviewed literature, even if it hadn't been blindingly obvious to anyone who knows anything about models. So if you want to bitch-slap someone it has to be either Santer or Schmidt, neither of whom seem to care about correctness. I tried to get through to Gavin but his only reply is that anyone who doesn't agree with him is wrong. Even Annan now agrees that it is a Bayesian calculation at best - after first trying every other wrong way.
From the error measurement POV each run is as statistically likely as each other run, limiting the runs used therefore just artificially limits the error estimate. Using frequentist stats is just blindly stupid. Even from the Bayesian perspective it is impossible to eliminate bias. Random sampling within the input errors gives you what climateprediction.net got: a big mess with either massive heating or massive cooling depending on the aerosol parameter used. But the reality is that the outputs are predetermined by the inputs anyway and the big variance around the mean is only due to the different guesses for the parameter fudge factors.
That anyone uses these models for policy indicates the generally poor level of useful education among policy-makers and the abject duplicity of modern science practitioners. It possible also indicates a declining standard among University graduates.
Jun 14, 2013 at 10:36 AM | Don Keiller
'ibbert's a bleedin' ginger 'e is. Yer carn't take nuffin' as gorspell from gingers, coz they is barmy as bedbugs.
@paulp
In the case of Climate Science it's more the madness of crowds.
Extraordinary Popular Delusions and the Madness of Crowds is a history of popular folly by Scottish journalist Charles Mackay, first published in 1841.
Freeman dyson is vastly more intelligent than any climate modelling technician. He has an intellect.
**
First, the computer models are very good at solving the equations of fluid dynamics but very bad at describing the real world. The real world is full of things like clouds and vegetation and soil and dust which the models describe very poorly. Second, we do not know whether the recent changes in climate are on balance doing more harm than good. The strongest warming is in cold places like Greenland. More people die from cold in winter than die from heat in summer. Third, there are many other causes of climate change besides human activities, as we know from studying the past. Fourth, the carbon dioxide in the atmosphere is strongly coupled with other carbon reservoirs in the biosphere, vegetation and top-soil, which are as large or larger. It is misleading to consider only the atmosphere and ocean, as the climate models do, and ignore the other reservoirs. Fifth, the biological effects of CO2 in the atmosphere are beneficial, both to food crops and to natural vegetation. The biological effects are better known and probably more important than the climatic effects. Sixth, summing up the other five reasons, the climate of the earth is an immensely complicated system and nobody is close to understanding it.
I am saying that all predictions concerning climate are highly uncertain. On the other hand, the remedies proposed by the experts are enormously costly and damaging, especially to China and other developing countries. On a smaller scale, we have seen great harm done to poor people around the world by the conversion of maize from a food crop to an energy crop. This harm resulted directly from the political alliance between American farmers and global-warming politicians. Unfortunately the global warming hysteria, as I see it, is driven by politics more than by science. If it happens that I am wrong and the climate experts are right, it is still true that the remedies are far worse than the disease that they claim to cure.
On the intolerance of Warmists:
You complain that people who are sceptical about the party line do not agree about other things. Why should we agree? The whole point of science is to encourage disagreement and keep an open mind. That is why I blame The Independent for seriously misleading your readers. You give them the party line and discourage them from disagreeing.
http://www.independent.co.uk/environment/climate-change/letters-to-a-heretic-an-email-conversation-with-climate-change-sceptic-professor-freeman-dyson-2224912.html
Dawkins: “We are all atheists about most of the gods that humanity has ever believed in. Some of us just go one god further.”
Models/Gods.
There's that religious connecton again!
JamesG, "So if you want to bitch-slap someone it has to be either Santer or Schmidt, neither of whom seem to care about correctness."
--------------
I'm keeping an open mind. The number could be more than two.
You'd need to bitch-slap a sufficiently large number and take the mean in order to be significant.
All of this verdammt statistics-ism (which requires holding onto the models in your mind--everyones' minds, oh the humanity!--under the assumption any of them represents "settled science", long enough to find fault, but only with the "ensemble") just to get to the point of saying the underlying physics, in all of the models, is wrong. Whew! And oy vey! And sonofa--.
This post tells me more about the barren wasteland that is quantum mechanics, than it does about the workings of the natural world. Prove that atomic electrons (electrons inside the "atom") even exist, as the electrons we know outside of the atom, Dr. Brown. Once you've assured yourself that they don't (and won a Nobel prize for what others have known for a long time now--read Dewey Larson's "The Case Against the Nuclear Atom"), then you will appreciate how hard it is to get climate scientists to accept that CO2 has nothing to do with global warming. The structure of the atomic electron cloud and the CO2 greenhouse effect are the same thing: They are both wrong physics, but "settled science".
The problem with calibrating anything against the actual climate record is that no such thing exists in any meaningful way. It certainly isn't Jones' dubious collection of old max / min temperature measurements from various locations of no particular relevance, no matter how many questionable statistical manipulations are applied to it in an attempt to make it so.
Murry Salby says as much in just two, maybe three sentences: linked to 58th minute
If you average two dozen models, their mean is a straight line at the decadal scale. If you average two dozen instrumental reconstructions over the 20th century, the average would preserve all observed decadal variability. Which means averaging model output = averaging noise.
Somewhat related - http://opinion.financialpost.com/2013/06/10/junk-science-week-unsignificant-statistics/