Bishop Hill Making fog
Dec 11, 2013 Climate: MetOffice 
Readers may remember Nic Lewis's paper demonstrating a major flaw in the UKCP09 climate predictions. In brief summary, the predictions are a weighted average of a series of virtual climates produced by the HadCM3 climate model, the weight each one gets being determined by how well it matches the observations. Nic discovered that the HadCM3 model was incapable of producing virtual climates that match the real-world climate as regards two key parameters - the climate sensitivity and the aerosol forcing. This obviously meant that the average produced is meaningless.
Nic's paper had a response from Julia Slingo which acknowledged that HadCM3 could not produce low-sensitivity/low aerosol forcing climates, explaining that this was an emergent property of the model. Nic noted that she was therefore implicitly accepting his core argument and I mentioned this in a blog post about the related UK Climate Change Risk Assessment.
Shortly afterwards I had an email from a press officer at the Met Office:
I noticed in your recent blog a line about UKCP09, as follows: “Unfortunately the UKCCRA is based on the UKCP09 climate predictions, which the Met Office has acknowledged contain a major flaw.”
The highlighted section is not accurate – no such acknowledgement has been made by the Met Office (unless you can point me to any evidence to the contrary which you think supports your statement). Could you please remove this reference from your blog?
This was a bit of a surprise to tell the truth. It was pretty hard to imagine how the Met Office could argue that the predictions were sound while acknowledging that the HadCM3 model behind it couldn't generate realistic climates. I therefore replied, trying to get some clarification of precisely what their position was:
As I understand it, the Met Office does not dispute that low CS/low aerosol forcing combinations are not possible with HADCM3. This is most likely the way the real climate system is, at least according to the current observational evidence. As the UKCP09 predictions are supposed to be a probabilistic combination of all possible CS/aerosol combinations, the fact that the model cannot sample the most likely combination is indisputably a major flaw. Prof Slingo has observed that this feature is an emergent property of the model, but this is only to recognise how the problem arose, not to dispute that it is an error. She has also stated that this feature does not create a bias, but I fail to see how she can conclude this without actually correcting the error and finding out what effect it has on the results.
I am happy to add a note to either of both of the BH postings clarifying the Met Office's position, but I would need to be clear what this is. Are you saying that it is possible to create a valid probabilistic forecast without sampling the full CS/aerosol parameter space and, in particular, without sampling the combinations of parameters that is most likely according to observations? I assume not. If the case is simply that the flaw doesn't create a bias (and is therefore not "major") then I would ask how you have reached this conclusion.
However, the reply the following day was evasive on these questions:
From your reply below it’s clear you make a few leaps of judgement to make your conclusion, and these aren’t leaps that the Met Office, or for that matter the IPCC, would necessarily agree with. Certainly they don’t justify the concept that we have acknowledged a flaw, let alone a major one – this is very much your own language and, as such, this should not be attributed to us in the way that you have.
I understand your argument here is based on Nic Lewis’s comment piece and so I note here that Nic himself has made it clear he did no claim there was an error in HadCM3 - in a response to an article on our blog, he said: “I didn’t claim that the HadCM3 model contained an error.” You can see this here (first comment).
Again, this reinforces the point that your statement is very much your own judgement call – not ours and, it seems, not Nic Lewis’s.
As a final point, HadCM3 was not the only model used for UKCP09.
With all this in mind, I’d appreciate if you removed the comment in question or reword it to something which more accurately reflects the situation.
Readers will note the complete absence of any attempt to answer my questions. Moreover, while the criticism had been carefully aimed at the UKCP09 predictions, here the Met Office was insinuating that I had implied a flaw in the underlying HadCM3 model (the argument is more that HadCM3 is unsuitable rather than that it is flawed per se). So this was an distraction by the Met Office. Readers might also like to read the rest of the Nic Lewis comment cited by the Met Office, which reveals a naked falsehood in the report they issued on climate sensitivity.
Similarly, the point about other models being used in UKCP09 was not the kind of thing you would have found in an honest response, as my reply made clear.
Thanks for your email, which arrived too late in the day for me to deal with before I went away on holiday. As I have indicated before, I am happy to make clear the Met Office’s position if you will make clear to me what that position is.
You note Nic Lewis’s comments about HadCM3, but I have been careful to say only that the UKCP09 predictions are flawed. Regarding HadCM3 we therefore appear to be in agreement and my blog posts are correct.
I note also your comments with regard to the use of other computer models in the UKCP09 project. I fear that you may have misinterpreted what your colleagues are telling you. As I understand it, the use of other computer models in UKCP09 is peripheral to the predictions, since they are only used in error estimates. Their use therefore does not in any way mitigate the problems caused by the structural rigidities in HadCM3.
I asked for clarification of the Met Office’s position in my previous email, but you have not responded to the question I asked. I therefore ask again: Are you saying that it is possible to create a valid probabilistic forecast without sampling the full CS/aerosol forcing space and, in particular, without sampling the combinations of those factors that are most likely according to recent observationally-based estimates? (For the avoidance of doubt, I am referring to values in (say) the ranges 1.25–2.25°C for climate sensitivity and -0.4 to -1.1 W/m2 for total effective aerosol forcing.) It appears to me that such a position is untenable and the UKCP09 predictions are therefore flawed. Your silence on the issue seems to represent a tacit admission that I am correct. You cannot say that I am wrong while simultaneously refusing to say why.
In a similar vein, I might ask why the Met Office has failed to withdraw its July paper when the vast majority of the problems Lewis identified in it have gone unanswered.
The Met Office cannot simply try to brush these problems under the carpet by refusing to acknowledge them. Should it do so it will undoubtedly be severely criticised and I therefore urge you and your colleagues to withdraw the UKCP09 predictions and the July briefing paper on your own initiative.
That was sent at the end of October but despite my sending reminders no reply was forthcoming. At the start of December I therefore wrote once more, indicating that I would publish the correspondence and let people draw their own conclusions. This finally elicited a response:
Sorry it has taken a while to respond, we do tend to get very busy in the winter months and I have to prioritise operational work, as I’m sure you’ll understand. With regards to our email exchanges, we appear to be going round in circles, so I thought I’d recap on how we got here.
As you’ll recall, this started when I asked you to change the following line in one of your blogs:
“Unfortunately the UKCCRA is based on the UKCP09 climate predictions, which the Met Office has acknowledged contain a major flaw.”
As I’ve said before, you are entitled to your own opinions but it’s inaccurate and misleading to ascribe your own opinions to someone else – which is what you have done in the highlighted line above. The Met Office has not said anything along the lines above, and that alone is enough to say that it is factually inaccurate.
In response, you’ve asked me numerous questions which don’t have any relevance to this straightforward request. Besides that, I understand Met Office scientists have spent a considerable amount of time over recent years dealing with similar inquiries from your correspondents, Doug Keenan and Nic Lewis in particular. In fact, Nic Lewis was here for an afternoon a few weeks ago and, as I understand, there has already been a robust, open and constructive conversation about many of the issues you raise. I think that is the appropriate forum for those discussions.
Having made my points as clear as I can, I’d once again ask you to remove or change the highlighted statement above and any other similar statements on the grounds that they are inaccurate and misleading. I think we’ve probably exhausted our respective positions, so I’d suggest that unless you have anything new to add, we bring a close to this particular line of correspondence.
If anyone is any the clearer for this response, they are cleverer than I am! Slingo has certainly admitted that the HadCM3 model can't do what it needed to do for the UKCP09 process. This is undoubtedly a major flaw. Ergo she has acknowledged the flaw.
Tiring of the word games, I decided that I would publish the correspondence anyway, but asked one last question:
Can you just give me one more detail please. You say that you think that my questions re whether the inability of HadCM3 to simulate realistic climates is irrelevant. Is this your personal opinion or the opinion of Met Office scientists (or perhaps just a corporate position)?
But unfortunately, this elicited an equally obscure reply:
We need to clarify here, as I wouldn’t want you to misunderstand or use my words out of context.
I said (as you can see in the email below): “...you’ve asked me numerous questions which don’t have any relevance to this straightforward request” [bolded for emphasis]. So to make it absolutely clear, I’m saying that your questions are not relevant to my specific request – which is on a point of fact, namely that you assert the Met Office has said something which it has not. I’ve gone into the details of why that is the case and why it’s misleading to ascribe your own opinions to someone else.
I don’t doubt that your questions do have relevant contexts and, as I understand, these issues have already been discussed at length between people more directly involved.
As I’ve said, my point is specific and on a point of fact. I am acting in my capacity as a press officer to point out factual inaccuracies and reasonably request that those inaccuracies are corrected.
To this I signed off as follows, copying Julia Slingo:
I have asserted that the Met Office has acknowledged that there is a flaw in UKCP09. I say this because Prof Slingo has acknowledged that HadCM3 cannot simulate low climate sensitivity, low aerosol forcing climates, which are the ones that observations suggest are most likely. She says that this is an emergent property of the model.
You have not disputed that she has made this acknowledgement. I therefore assume that you are disputing that HadCM3's inability to simulate these kinds of climates means that the UKCP09 projections are flawed. This is, as your scientific colleagues no doubt realise, is a completely absurd and entirely indefensible position. It is hard to believe that anyone within the Met Office would seriously suggest that a weighted average of climates excluding those most like the real climate has any usefulness as a predictive tool. As if to confirm this conclusion, you have signally failed to answer my questions on the matter.
I assume then that your real objection is that your colleagues are embarrassed by this public airing of their errors. That really is too bad. I reiterate what I said in my earlier email: the duty of the Met Office is to withdraw the UKCP09 predictions of your own volition as soon as possible. To allow both public and private sectors to continue to spend on the basis of erroneous predictions would represent a serious breach of the Met Office's duties towards the public.
I will add a note to my earlier post as follows:
The Met Office have acknowledged Nic's points about the flaw in the UKCP09 predictions but ask me to make it clear that they deny that they have acknowledged a flaw. I don't know what they mean either.
Bishop Hill
In response to comments by Richard Betts, Nic Lewis writes as follows:
Fig. S2 of Harris 2013 shows nicely that the HadCM3 PPE can't generate low ECS, low aerosol forcing combinations. Aerosol forcing is about 0.3 W/m2 more negative than ΔQ_noghg, so the AR5 best estimate of aerosol forcing (-0.9 W/m2) corresponds to ΔQ_noghg = -0.6 W/m2. The only place where the
HadCM3 PPE gets close to the AR5 best estimate of aerosol forcing, let alone below it, is at ECS > 3 C.
The small diamonds show the effect of perturbing the aerosol module parameters, the big diamonds being the default settings. Most of the variation in aerosol forcing comes from perturbing the main atmospheric model parameters, which also changes HadCM3's ECS. The dotted lines are meant to show uncertainty bounds, but I think they are wrong - they should all converge at low ECS.
;)
Reader Comments (128)
In the spirit of tight moderation, I hope we don't move into discussing Nic Lewis' work, a distraction from the post's argument.
That argument has been just reinforced by the updating with the graph: the MetOffice's model(s) breaks down at low sensitivity value (low as in nearer to zero), as it can't do a thing without very low aerosol feedback (low as in "low temperature").
In fact the diamonds mysteriously stop at 2.1C, perhaps indicating that the MetOffice cannot model 20% of the 1.5C-4.5C IPCC AR5 WG1 climate sensitivity range.
ps I do not expect Richard to be able to be forthcoming on the topic, unless he is ready for his own rendition of this.
Omnologos
Agreed. I should really snip Richard's comment entirely, since as far as I can see it is entirely off topic. Nic's opinions on CS or AF are irrelevant to the question of whether HadCM3 can sample the whole CS/AF parameter space.
Since it's Richard I will leave it, but responses will be snipped.
'"the UKCP09 climate predictions, which the Met Office has acknowledged contain a major flaw." simply is not true.'
So the Met Office didn't acknowledger the climate predictions contained a major flaw. Now that that's out of the way, can we ask if there was, or wasn't, a major flaw in the UKCP09 predictions?
Fair enough, Bish. But it would be interesting (to this reader at least) to be able to read Nic's response to Richard, should Nic be minded to pen one. I note that Nic did in fact respond on James Annan's blog to the passages Richard quoted. (In addition, I'd be most interested to know whether Nic has addressed the comment on that thread by Rob Dekker.)
Hi Richard
Thanks for clarifying what it was you were disagreeing with. I agree James is a fair choice so far as academic climate scientists go. However, I've looked into the area of estimating climate sensitivity from estimates over the instrumental period in more depth than James has – he has published more in the area of paleoclimate-based estimation.
James was talking about a blog-published sensitivity estimate I came out with a year ago. I rebutted his comments in a detailed, factual response on his blog, here. In a nutshell, he was factually wrong about my heat uptake figure, and I was simply using the IPCC AR5 SOD satellite-based mean estimate of aerosol forcing – which was consistent with good quality inverse estimates.
As you know, the Nature Geoscience energy budget Otto et al (2013) study (the one Met Office hates) that I was involved in, along with fourteen lead authors of AR5 chapters relevant to estimating climate sensitivity reached pretty much the same conclusions as my last December's blog energy-budget study. In fact, my study gave an (unstated) transient climate response (TCR) estimate of 1.3 C, dead in line with the Otto et al estimate using the latest decade's data and only a few percent below the Otto estimate based on the longer 1970-2009 period, which the Met Office regards as preferable. My climate sensitivity estimate was a bit lower than the Otto et al estimates (1.6 C vs 1.9 C estimated over 1970-2009 and 2.0 C over 2000-09). But for technical reasons the use of GCM derived forcing estimates is likely to have biased up estimation in Otto et al., and the heat uptake estimate it used was top of the range, particularly over 2000-09.
You actually wrote that "Met Office (and indeed everyone else as far as I am aware)" thinks my estimates of aerosol forcing and climate sensitivity are unrealistic since the assumptions behind them don't seem to stand up. Have you read studies like Aldrin et al (2012) and Ring et al (2012)? And what unwarranted assumptions exactly lie behind Lewis (2013), my study published in J. Climate? These studies all produce aerosol forcing and climate sensitivity estimates very much in line with my energy- budget based one you criticise. All these studies are based on multi-decadal warming over the instrumental period, which as you probably know AR5 regards as providing the most reliable estimates of sensitivity. The studies form their own estimates of aerosol forcing from how well modelled latitudinal evolution of temperatures fits observations as the strength of aerosol forcing in the climate model used is varied. Unlike HadCM3, the models used permit aerosol forcing to be varied independently of climate sensitivity.
I am aware that AR5 showed higher climate sensitivity and TCR estimates from various studies that were also based on warming over the last 100–150 years. However, I have analysed these studies in more depth than in AR5, and have found serious flaws in all the studies that gave higher (more in line with GCM-based) sensitivity estimates. See here.
Returning to sensitivity estimates using the robust energy budget method, I'm happy to argue from the AR5 best estimate of aerosol forcing of -0.9 W/m2 – and expert judgement reflecting both satellite-observation and model-simulation based estimates of aerosol forcing. Using the AR5 unadjusted estimate for aerosol forcing as well as for heat uptake, the resulting aerosol forcing - climate sensitivity combination remains outside the region sampled by the HadCM3 PPE used for UKCP09.
It seems clear from my discussions at the Met Office that biggest issue is whether one places more reliance on observationally-based estimates or on simulation results from complex models. Impressive achievements though AOGCMs are, I think giving precedence to observations over models is a better physical science approach. The Met Office philosophy seems to be the opposite.
If I'm understanding Richard correctly he seems to be insinuating that it is OK for the HadCM3 PPE not to sample low-CS/small negative AF climates because such climates are unrealistic. But surely the fact remains that had they been sampled they would have been heavily weighted in the final reckoning because they match observations.
Perhaps if you could answer the question I keep asking you and your colleagues, namely do you believe that it is possible to create valid set of projections without sampling the full CS/AF space?
Bishop Hill
Well, yes, of course - the full CS/AF space is infinite! We should focus on the region of the space which is realistic.
Your argument that
seems strange to me, because it appears to suggest that it would OK to get the right answer for the wrong reasons. i.e.: it suggests that wouldn't matter why the model agrees with observations - even if it were because of the cancellation of two unrealistic processes, then this would be fine. If this is what you are saying then I disagree - it is also important that the physical processes in the model are realistic.
I'll sign off for now, as it's the weekend and I've got other stuff to do (like get stuff sorted for Christmas!). However, I hope this has helped explain why the Met Office does not consider it a major problem that the HadCM3 PPE does not go into the low CS/weakly -ve AF territory, and hence why my colleague in the press office pointed out that it is incorrect to claim that "the Met Office has acknowledged [that the UKCP09 projections] contain a major flaw."
Cheers
Richard
Precisely - the car explodes in a crash but since that's what the car does, it's no flaw.
And if you have a camera always pointed due North you can report that the whole world is to your North because that's what the camera always sees. There's no flaw in your report.
Ps this Xmas Richard can serve chicken to the family because the chicken knew it was a chicken and the family can see it was a chicken, therefore it's ok to call it a turkey.
BH
"If I'm understanding Richard correctly he seems to be insinuating that it is OK for the HadCM3 PPE not to sample low-CS/small negative AF climates because such climates are unrealistic."
That is unrealistic according to HadCM3 and the Met Office modellers, of course.
So far as the actual climate system goes, the lowish climate sensitivity/smallish negative aerosol forcing combination is what observations seem to imply is the most realistic one.
Richard, when you return, are you going to offer any evidence that contradicts this assertion, which I've cited several studies in support of as well as calculations based on data in AR5?
Summary:
They acknowledge the major flaw.
They just haven't said "We acknowledge a major flaw".
Clear now?
Nic
Doesn't that completely negate the point of doing a perturbed physics ensemble in the first place? If the point is to work out the best set of input parameters by comparing outputs to observations then why would you want to exlude some outputs on the basis of a comparison to a run of the model, presumably with best-estimate inputs.
I have to say that I admire RichardB's loyalty and polite persistence but if it looks like a duck, walks like a duck and quacks like a duck ...
Bish
why would you want to exclude some outputs on the basis of a comparison to a run of the model
I don't think they are excluding anything. The next set of diamonds to the left would very likely go so deeply down as to make the meaningless of the whole thing apparent.
Just like you can't use the Standard Model before Planck's epoch, you can't use the MetOffice model(s) before an ETS of 2C.
ps "before"="below" for the Met Office employees in the audience :)
Seems to me that Richard Betts is quite happy for the Met Office to get the wrong answer for the wrong reasons. Getting the right (scientific) answer is not something the Met Office apparently aspire to. Getting the right (political) answer, no problem.
PR 1 - 0 Science. Again!
BH
" If the point is to work out the best set of input parameters by comparing outputs to observations then why would you want to exlude some outputs on the basis of a comparison to a run of the model, presumably with best-estimate inputs."
I don't think the Met Office scientists deliberately excluded any AF/ECS combination regions. In fact, Glen Harris and David Sexton seem to have tried quite hard to explore the parameter space as widely as they could. The problem is that HadCM3 has a structurally rigid relationship between aerosol forcing and climate sensitivity, which they didn't succeed in overcoming despite perturbing lots of key parameters. I don't think they really focussed on the implications this relationship had for the comparison between model simulations and observations. It makes that comparison of little use - in fact positively misleading. Unless of course one believes, as some Met Office scientists may, that in the actual climate system the AF/ECS relationship must be close to the region that the HadCM3 PPE can explore.
thanks for confirming my suspicions Nic.
Unless of course one believes, as some Met Office scientists may, that in the actual climate system the AF/ECS relationship must be close to the region that the HadCM3 PPE can explore.
Is Josh still around? A nice old joke might be in need of its own climate version...
Lost keys and lampposts...
Omnologos: Great minds think alike but perhaps sometimes in reverse. As I read Nic, Rich and the Bish on Friday I was also thinking of the drunkard, his lost keys and the lamppost. But being a good empiricist and a programmer of software models who knows what a bad one looks like (ahem), I would never ascribe more light to HadCM3 PPE than to real-world observations. The perversity then becomes that this collections of drunks lost their keys under the lamppost but insisted on looking for them in the dark, led by a delinquent teenager. Can the accusation that they didn't actually want to find the low sensitivity that saves humanity's bacon be any surprise as we all endure the big hangover?
Richard
Keep in mind what Gavin wrote when challenged by John Tierney on the relationship between models and observations
http://omnologos.com/more-on-realclimates-unfalsifiable-models/
this subject appears to have been raised from the expectation that some short term weather event over the next few years will definitively prove that either anthropogenic global warming is a problem or it isn’t. As the above discussion should have made clear this is not the right question to ask. Instead, the question should be, are there analyses that will be made over the next few years that will improve the evaluation of climate models?
I surmise the climatists at the Met Office are in the business of working around lampposts too.
Very good piece from April 2008 Maurizio. It's not, as Nic says, that the AOGCMs aren't a worthy intellectual achievement in their own way. But a wee bit like Narnia the lamppost stands for the proper junction between the two worlds. Those of us shivering in this one are indeed waiting for the big thaw. (The analogy isn't perfect, please don't push it!)
@Richard Betts: "it is also important that the physical processes in the model are realistic".
Can you prove that your idea of realism of physical processes is correct? Have you included all possible mechanisms that cause a negative feedback on the global temperature? Extra negative feedbacks would make a smaller climate sensitivity realistic than you might at first expect.
Nic
Yes, of course, my comment was loosely phrased. Nevertheless, Richard has tried to justify the lowCS/small negative AF climates not being sampled. So while there was no intent, he is saying that these virtual climates being overlooked (is there a better word?) is not a problem.
It is not a problem because the Met Office choose to practice advocacy rather than science. But THAT is a problem. It is THE problem.
Hi Nic
Your approach seems to rely on the assumption that the future behaviour of a non-linear system (i.e.: the climate) under new conditions can be predicted with high precision on the basis of the statistics of its past behaviour under a rather limited set of previous conditions. I think this is a huge assumption. ECS is the long-term response of the climate system to forcing from a doubling of CO2, which has not previously been seen - in the instrumental period we've only been able to see short-term responses to a rather smaller forcing.
In a non-linear system with lots of feedbacks, assuming that a small sample of past behaviour is representative of future behaviour under a larger external forcing requires something of a leap of faith.
Aldrin et al (2012) do highlight the uncertainties associated with feedbacks and non-linearities.
Cheers
Richard
Hi Richard
You're not addressing the main issue here, which has nothing to do with the very long term response (equilibrium as opposed to effective climate sensitivity, if you like) or whether it can be accurately estimated either directly from historical observations or by model simulations, whether or not constrained by observations.
The objection to the use of the HadCM3 (strictly, mainly in its HadSM3 slab ocean variant) PPE that I was voicing is that it essentially does not sample combinations of smallish aerosol forcing and lowish climate sensitivity. HadCM3/SM3 does not in fact exhibit a hugely nonlinear response at up to the doubling of CO2 levels specified in the definition of ECS. So there is no reason to doubt that this region does correspond to where, according to the AR5 best estimates of forcings and heat uptake, the best fit with observationally-based evidence lies (albeit some of that evidence requires use of models and is not fully observational).
Even if non-linearity or time dependence of sensitivity were in point, the issue would still arise in relation to regions in aerosol forcing - TCR space, which is unaffected by such problems and is also more relevant to projected warming towards 2100.
So my objection remains valid. Unless of course one regards HadCM3 as more informative about the real climate system than historical observational evidence. If that is the Met Office's view, they should be upfront about it and not claim that the UKCP09 probabilistic projections are observationally constrained.
Happy Christmas,
Nic
Hi Nic
I already addressed the main point earlier, which is that it is being claimed that the Met Office said something it did not say. We disagree that the low ECS/low AF point is a "major flaw in UKCP09". You and the Bish clearly think that it is, and you're entitled to your opinions, but it's not correct to project that view on to the Met Office.
Happy Christmas to you too!
Richard
That was Episode #476 of Richard Bett's "River of Fire" impersonation.
RB - So a relevant and important part of the sample space is not considered (in fact the model is not able to consider it); that part of the model space corresponds to observable phenomena. You agree that the model is not able to consider it. If, as seems increasingly likely, the real solutions are in that space, then your model can't describe them.
PR is one thing Richard, but your credibility is at stake here.