Bishop Hill On tuning climate models
Jan 30, 2015 Climate: Models I had an interesting exchange with Richard Betts and Lord Lucas earlier today on the subject of climate model tuning. Lord L had put forward the idea that climate models are tuned to the twentieth century temperature history, something that was disputed by Richard, who said that they were only tuned to the current temperature state.
I think to some extent this we were talking at cross purposes, because there is tuning and there is "tuning". Our exchange prompted me to revisit Mauritzen et al, a 2012 paper that goes into a great deal of detail on how one particular climate model was tuned. To some extent it supports what Richard said:
To us, a global mean temperature in close absolute agreement with observations is of highest priority because it sets the stage for temperature-dependent processes to act. For this, we target the 1850-1880 observed global mean temperature of about 13.7◦C [Brohan et al., 2006]...
We tune the radiation balance with the main target to control the pre-industrial global mean temperature by balancing the [top of the atmosphere] TOA net longwave flux via the greenhouse effect and the TOA net shortwave flux via the albedo affect.
OK, they are targeting the start of the period rather than the end, but I think that still leaves Richard's point largely intact. However, Mauritzen et al also say this:
One of the few tests we can expose climate models to, is whether they are able to represent the observed temperature record from the dawn of industrialization until present. Models are surprisingly skillful in this respect [Raisanen, 2007], considering the large range in climate sensitivities among models - an ensemble behavior that has been attributed to a compensation with 20th century anthropogenic forcing [Kiehl, 2007]: Models that have a high climate sensitivity tend to have a weak total anthropogenic forcing, and vice-versa. A large part of the variability in inter-model spread in 20th century forcing was further found to originate in different aerosol forcings.
And, as they go on to explain, it is quite possible that a kind of pseudo-tuning - I will call it "tuning" - is going on through the choice of aerosol forcing history used (my emphasis):
It seems unlikely that the anti-correlation between forcing and sensitivity simply happened by chance. Rational explanations are that 1) either modelers somehow changed their climate sensitivities, 2) deliberately chose suitable forcings, or 3) that there exists an intrinsic compensation such that models with strong aerosol forcing also have a high climate sensitivity. Support for the latter is found in studies showing that parametric model tuning can influence the aerosol forcing [Lohmann and Ferrachat, 2010; Golaz et al., 2011]. Understanding this complex is well beyond our scope, but it seems appropriate to linger for a moment at the question of whether we deliberately changed our model to better agree with the 20th century temperature record.
They conclude that they did not, but effectively note that the models that find their way into the public domain are only those that, by luck, design or "tuning", match the 20th century temperature record.
In conclusion then I conclude that Lord Lucas's original point was in essence correct, so long as you conclude both tuning and "tuning". Richard's point was correct if you only include tuning.
Reader Comments (74)
And the most convincing, highest quality bullshit is produced by those who sincerely believe their own bullshit.
Martin A There are a lot of people who would argue with that, without realising they were proving you right.
As other have said tuning climate models is purely circular. Tune your model to temperature databases that have themselves been tuned to fit "the theory". Evidence of dodgy adjustments comes to light nearly every day.
It just amazes me that climate modellers have so little awareness of their inbuilt confirmation bias as to still think they are modelling the Earth's climate.
The models are extremely clever but they have little to do with our climate and how it will change.
It's interesting that the man who shouted loudest about quantum electrodynamics was Paul Dirac, who was autistic.
He couldn't understand the ' keep quiet, we're all in this together, circle the wagons, cognitive dissonance affected attitude of his colleagues. We see this collective defensiveness in climate science, especially toward evil deniers.
##
Dirac almost turned down the Nobel Prize because he didn't want the publicity. He only accepted it when his colleagues pointed out to him that rejecting the prize would generate far more publicity than accepting it. When he arrived to collect the Nobel Prize, he caused confusion and panic by sitting quietly in the railway station's waiting room while the welcoming committee of grandees lined up on the platform became increasingly worried about his whereabouts.
http://www.newstatesman.com/society/2010/11/dirac-autism-autistic
Q: Sire, Dear noble sirs... just beyond the square, the town stables are full of horses from all the lands near and far. Would it not be prudent to travel to the stables and look in the horse's mouth and count the teeth instead of all this long winded mathematics modeling the number of teeth in a horse's mouth?
Reply from Lord Bob (snipped due to vulgarity)...
A lot of insightful comments above. I'd just add my two penn'eth. First model I built was based on a discrete differential equation analysis and intended to predict the accuracy of a measurement on the diffusion of a simple mixture of two gases. It was based on sound and well understood predictable physics. It was in the early 70's and took overnight to run to completion on what was then a leading edge mainframe IBM machine, the days when one delivered punched cards to the computer technicians and received output on reams of paper.
When I think about "climate" models I just shake my head, the sheer arrogance of thinking we can model a chaotic system on which we have so little concrete knowledge never mind accurate measurements is unbelievable.
whmph ....the sound of things going over my head.
There is so much I don't know & so much I don't understand.
I'll leave modelling to you guys.
But I will step right back and ask a general question. Is this the way the whole process works ?
- Monday - you prove that your model that incorporates "CO2 causes disaster" works
- Tuesday - the world stops all CO2 emissions.
- Wednesday - the climate returns to "normal" ..ie all the significant indicators like ice levels, sea levels, amount of extreme weather, flooding etc. for the next FUTURE 50 years all stay within "natural variation" just like they have for the last 50 years.
.... or is it a bit more complex than that ?
Like that CO2 hangs around so long that you'd wind up using a geo-engineering solution, even if you stopped emissions on Tuesday, just the same as if you waited 5 or 10 years to stop emissions.
No, the models are "tuned" to the AGENDA.
I will repeat a comment I made at Lucia's that begins to explain how I have tuned models in other disciplines, and why we actually want more tuning.
"Tuning is a systematic process where you vary uncertain input parameters to minimize the error between model outputs(M`) and real world observations (M).
For example.
You have a model that is
M` = f(a,b,c,d,e,f,g,h,i,j..) Where M` is the metric of interest ( there may be other outputs ) and {a,b,c..} are uncertain/free parameters that you can vary.
each parameter has a ‘range’ of what are considered to be possible values and you can construct your model to vary these parameters in a systematic fashion to effectively find those settings or range of settings which minimize the difference between the model output M` and the observations M.
A simple example from engineering. We had to create a high fidelity model of the F-5 flight control system with very little documentation ( we had left Northrop and could not get the documents) Well, we had system diagrams and we had flight test traces. The system diagrams told us what basic building blocks there were in the system ( the physics in a metaphorical sense) But what was missing were key specs on certain devices ( accelerometers for example ) So you coded up the model and threw in numbers for all the missing device specs. Guessing in an educated fashion.
Then you matched model output to the flight test.. And after a few weeks of this you could say that the accelerometer HAD to provide a measurement within x% of the true ‘G’s’ in order to match the flight test. This was proof that the original component delivered these specs even though no one had access to the actual spec sheets. Actually, for some folks it was better proof than a spec sheet would have been. Given the flight test data and the block diagram one could interatively solve for the device performance specs.
This worked because
A) we had reliable data. (flight test)
B) we had a full representation of the physics.
C) tuning is basically just rearranging the math and givens.
or to put it more precisely, this works subject to your assumptions on 1 and 2.
Given you have certain test data, given you have a full physical understanding, you can use modelling to resolve uncertainty in your input parameters. Inverse problem style.!! Instead of predicting the output one takes the output as given and predicts the inputs. durr.
Now the cool thing here is that the model actually output more than one metric. It output M1,M2,M3, M…. and so you could tune on M1, and then check on M2,M3,M4 etc. Of course if M2,M3,M4 can be derived from M1, that doesnt tell you much, but if they are in different physical measurement units that improves your confidence that you’ve done things right. In GCM world you can tune on temperature ( a small segment of the data) and check other outputs.Bonus.
So what do you know when tuning fails.
Suppose you had a model that predict global ice and your one tunable parameter was “initial ice cover”. and you varied this from 0% to 100%. and suppose that none of these settings led to anything close to the right answer. Well, that would be good evidence that your physics was wrong or incomplete. Even failed modelling experiments give you information. yes Virginia models do give you observations.!!
In this case the observation is that your physics is wrong. I always laugh when Sceptics say that models dont produce evidence or data. And then they use the FAILINGS of GCMS as evidence that they dont capture all the physics. err, well we will leave that skeptical logical lapse for another day.
Moving on....
Suppose you can reproduce M or have a small error between M` and M. What’s that show you?
It shows you that Given acceptance of your observations M, you can assert a complete relevant physical understanding of the system that is within the uncertainty of your input data. That is, there is possible world where your physics is true. Or rather, given the uncertainty of the input data your physical understanding can’t be rejected. It is a good explanation, subject as always to improvement or replacement.
Back to GCMs. There are a couple of interesting points to make.
First is that GCMs have huge metric spaces. We could look at temperature, land ocean contrast, spatial variability, SST, MAT, SAT, wind, clouds, precipitation, sea surface salinity, etc. Tuning to one of them and getting the others correct actually gives us confidence that the physics is correct. But If we tune to temperature and get the land ocean contrast wrong, Then WTF .. danger will robinson.
There are a few low resolution GCMs where one could do the kind of tuning mentioned above. Example would be FAMOUS.
http://download.springer.com/s.....8;ext=.pdf
However, Tuning a GCM in this manner in intractable. In short,
you can’t afford to vary all the parameters to actually tune
the model to output parameters. The best you can do is “hand tuning” Someone on the team found a range of settings that get’s you close. dont touch those knobs fool. touch this knob or that knob, but leave the others alone.
One approach to improving this situation is to create an emulation of the GCM. James has some work on this
http://classic.rsta.royalsocie...../2077.full
So.
1. Do the teams vary all the parameters in a systematic fashion to minimize the difference between the output metric and the observations. No. They don’t Tune.the models are not TRAINED.
2. Are there knobs they twist in a limited fashion to give “sane” results? Yes.
3. Do all teams twist the same knobs and look at the same metrics? No. Some, for example, may only do a sanity check that they get 1850-1890 “correct”.
If they ACTUALLY tuned and actually matched observations that would be a GOOD thing as it would demonstrate a good physical understanding within the limits of input uncertainty/structural uncertainty. It would show a consistent physics capable of explaining things within known uncertainties.
One doesnt want less tuning, one wants mo better tuning.
The truth is that the temperature data has been "tuned" to fit the predictions of the models.
Eugene WR Gallun
MikeN, If you have been delivered a million radios all playing whatever they are tuned for randomly - you won't hear what's going on.
Now imagine you throw out every radio that isn't playing the Archers - suddenly you have Radio 4.
So the next radio through the door - will it tell you what's going on down the pub in Ambridge? Or is that Bull?
Throwing out the radios is tuning the remaining radios on Radio 4 without any real understanding of what wavelength it's on.
That's how climate models work.
I think models in general and GCM in prticular are very useful, especially when they fail, because that tell us what is that we don't understand well. But I feel that this is use is not very welcome in climate academic circles.
mosher, you needn't be told this but tuning works differently in the domain of hypothesis testing as opposed to working and workable systems.
Moshe: "In this case the observation is that your physics is wrong. I always laugh when Sceptics say that models dont produce evidence or data. And then they use the FAILINGS of GCMS as evidence that they dont capture all the physics. err, well we will leave that skeptical logical lapse for another day."
I have read sceptic blogs for a long time now and have never heard, or read, anyone who said the models don't produce data. I've heard plenty of people say the output of models is not evidence of anything and point to the data from them not being consistent with the observations in the real world as proof of their lack of skill. But not one person I have ever met, with the single exception of SM it seems, has ever but up the bizarre argument that models don't produce data. Of course they produce data, but that data isn't evidence of anything until confirmed in the real world.
So, if you take out the word "data" from your assertion that sceptics say that models don't produce it your logical non-sequitur disappears. I hope this helps.
A view from evolutionary biology-
It seems that what we really have here is an ecosystem full of models subject to natural selection. Models proliferate. Models that don't perform fall by the wayside, others are tuned to reflect longer periods of real observations, until after a few generations some survive to emerge which seem to have some "skill" in representing the temperature history we have observed. . But this process definitely doesn't mean that the models' equations or sensitivities are actually correct, or even that the logic driving them is correct. It only means that those models are better adapted to the model ecosystem in which they live. The ONLY test which is of interest is in predictive skill, and at present that's where they have all been falling down..
Our revised and improved predictive bridge building computer model is now available. While, yes, it is still true that none of the functions within are capable of calculating tensile strength of metal alloys or are capable of calculating the hardness of the cement which we realize makes up the greatest percentage of our projected projects I am happy to now announce that recently with much surprise our senior most and glorified 'scientists' have increased our confidence in our product from ninety percent to a confidence of an unprecedented 95 percent. This is a true milestone and I am certain that now even the worst skeptics will realize that our revised and improved bridge building computer model should be utilized and all other companies and govt. agencies should be disbanded and pushed to the side.
some survive to emerge which seem to have some "skill" in representing the temperature history we have observed.
...
Jan 31, 2015 at 10:14 AM mothcatcher
My climate model is simply a table in a spreadsheet. It reproduces the temperature history we have observed with complete accuracy. Clearly a strong candidate for survival.
Should this give me confidence it can reliably predict future climate?
If you understand the system being modelled then there will be no need to tune it. If you are having to tune the model then it follows that you clearly do not understand the system. If you do not understand the system how do you know which parameters to tune and what is the subsequent value of the model's output; especially when more than one parameter is being changed at a time? You have a 'bubbles in wallpaper' problem; flatten one bubble and a new one will pop up in some other unpredictable part of the wallpaper. You will always have bubbles in your model until you understand and cure what's causing them. If you don't know what's causing them or even how many you have, then what is the value of the model?
The value of the model? To paraphrase head cheese Schmidt, some facets of the model are skillful.This seems to align nicely with the goal of climate models, which is to teach us something about the climate.
Tuning seems rather boring conversation when viewed in this context.
Does tuning to a chaotic past lead to a chaotic future? If yes then the process is useless, if no then tuning has changed the system from chaotic and is also useless for predicting a chaotic system's future.
ghl
Betts' favourite trick. He throws out strawmen in an effort to divert the conversation away from the more serious failing in UKMO doctrine.
Fact is their models are useless, expensive toys which provide them lots of pretty and useless (except for PR) pictures. Their one model that appears to provide a semblance of usefulness is the short term (12hr) part of their weather-climate model. It can provide reasonable forecasts in the short term but only if the chaotic system has settled into one of its' many short term attractors.
It's time to close the "climate unit" of the UKMO and use the money on something realy useful such LTR electricity generation.
I find I have to agree.
The 'climate forecast', if it was ever useful or believable, remains the same. By their own metrics, their job is done.
Mosher, so how did RealClimate tune a model to produce a 3C output?
What should replace the Met Office?