Thursday
Jan102013
by 
Bishop Hill
Bishop Hill Spot the difference
Jan 10, 2013 Climate: MetOffice Climate: Models I'm still suffering. Even whisky isn't working. It must be serious.
In the meantime, Paul Homewood has found something interesting about the Met Office's forecasts.
Reader Comments (216)
Roger Longstaff says:
"The only way to judge the forecasting capability of MO models is to compare their predictions at the time (with the best model and up to date data at that time) with previous and later predictions. If I am correct in this then the worst that the MO can be accused of is confusing terminology."
I am not so sure about this, I think it depends on how the models run. I have been doing some back reading on these models and it is not clear to me whether this model includes a stochastic compoent in order to efficiently handle non-linear terms.
If the model is purely deterministic then for a given set of intialisation data/parameters etc re-running the model should give the same output every time. If stochastic perturbation is used in some of the calculations (eg to make non-linear terms more efficient in computation, or handle phase transitions , the result of which can be quite non-linear, with some uncertainty) then re-running the same model with the same parameters/initialisation etc but a different random number seed would give a different result each time. If this is the case, then it would be possible to run several cases "realisations" of the model and then select the "realisation" that best matches the data record for subsequent years. Using this approach would give a false sense of the validity of the model.
This is why I have asked several questions of Richard Betts further up this thread.
Before/after
(Found somewhere on the internet)
For those of you who don't know anything about stochastic models who don't follow the points I have been posting here, perhaps an explantion will help.
If the Met Office model is purely deterministic then for a given intialisation/parameters you always get the same prediction no matte how many times you run the computation. This would be like fitting a trend line to some data, the trend line is your model and by extrapolating into the future you can make a prediction which could be compared with the actual newer data (that wasn't used to fit your model).
Alternatively imaging a purely stochatic model, one created by random number process. If that process was run once, I would get a random sequence of tempratures for subsequent years. They might be a very poor fit to the actual temperatures. But if I ran that random process say 1 million times (a million "realisations" of the temperature predcition over say 30 years into the future) and selected the result that best fit the actual temperatures in later years and presented it as my model it would look remarkably good. Except it would have no predicitive capability whatsoever (other than the mean, or long term trend I might have added to it as a parameter).
These types of accept/reject approaches are used in many disciplines (including mine) to fit or condition to data where the conditioning problem is highly non-linear or no formal physcial model exists. The trouble is they are totally inappropriate models in the absence of conditioning data, unless used as a set of realisations. In other words, I cannot cherry pick one realisation and show it, I would have to examine all the realisations (is this what they mean by an "ensemble"?) together, which would give me an understanding of the variability around my model (note it is flase to refer to this as the "uncertainty", as that implies the fundamentals of the model itself are correct).
Hope I haven't confused people more, but I think this is a very important point about exactly what type of model it is that is producing the output, and how it runs computationally.
Richard Betts: You say:
Does this mean that at no stage during the development of the new model did anyone ever go back to tweak the input parameters on the basis of the output?
because if they did use the output to fine-tune the model, then that should not have increased your confidence in its ability to forecast at all. Merely to hindcast.
@ThinkingScientist
My understanding is these are stochastic and the red shaded parts of the Met graphs show the distribution of the spread of modelled outcomes over he period.
@Richard Betts,
Many thanks for your contributions. May I ask you a question, not directly related to the subject at hand.
The panic over rising global temperatures suggests that there is a known optimum global temperature. What is it? And if we don't know that it is, how can we possibly say that a rise in temperatures is bad for the planet?
[NB. Personally, I don't see HOW a aggregated global temperature can have any meaning at all. But that's just me, I guess]
Gecko,
If the models are stochastic, then do they show as their prediction the mean of all of the realisations ( which would be correct) or a selected realisation?
Matthu, I think "tweak the input parameters" is grossly understating what happens.
All known data about the climate to date are INPUT to the model - of course, why wouldn't they be?
Clearly the model has many modules that are based in physical rules, and these are of necessity tuned to observations. This is perfectly normal science - in fact, this is almost a definition of science. The relative contributions of these modules can only be determined by measurements, even if the absolute effect of some specific "forcing" can be determined theoretically.
You may as well accuse physicists of "tweaking the value of an electron volt" when they determine the value to the bast accuracy that they can. That's just science.
Steveta says: "Clearly the model has many modules that are based in physical rules, and these are of necessity tuned to observations."
I disagree. This is (should be/is described as) a deterministic forward model, not a heuristic/empirical model. I could create a series of empirical relations (eg complex trend interactions) from observations with no knowledge of the physics and use them as a predictive model. It may be a good representation of the actual physics, but it would not be science. That's statistics, not physics (science). A physical science model is quite different from a trend line model, even though they may give similar predictions.
These models are paraded as solving the physical problem of forward modelling of climate, not as empirical or heuristic descriptions based on observations.
@ThinkingScientist
I think that is the median. So they should be intrpreted to display a spread of where 90%, say, of iterations fall.
@ThinkingScientist
I think that is also why there is a running battle over the application of the phrase "confidence intervals" with respect to climate models. That of course being something entirely different to the distribution of model iterations.
Gecko,
We really need Richard Betts to formerly confirm what they are showing.
On the point of the median, do they show for each year the median (P50) independently of the other years, to draw the median line, or do they show the (valid) realisation which is closest to the P50? In a non-linear response or complex system these two ways of representing the result could be quite different. Eg it would be quite possible for the median from each year independently to produce an apparent flattening in the prediction for 20 years, but no realisation from the set of realisations would actually exhibit that behaviour due to eg autocorrelation across the years.
To understand this, imagine rolling a 7-sided dice, with each roll representing one year. Put these together to form a single realisation. Repeat the process until you have lots of realisations. For any year, the mean (and median) will be 4 (which is why I didn't use 6-sided dice!) and the result of plotting the independent yearly P50 from the set will be a flat line. However, if you select a realisation it will have more local jitter and will only show flat segments by chance over short intervals.
(For the pedants, I am aware that the singular of dice is die, but dice reads better).
Japanese whisky is just fine.
Anyone interested in the history should look up Masataka Taketsuru, his studies in Scotland, and his taking of the brave bride Jessie Roberta Cowan. She was a woman with spirit.
From the nature of the plot I would say the former (compare the time series nature of the white with the extremity of the red shaded regions).
As such you would read it as the median interation to any particular point in time.
Has this cessation of the rise in temperature anything to do with the removal of temperature-recording stations from Arctic and high-altitude areas, and an increase in reliance on such stations in UHI areas? When did this policy start, and when was it concluded?
Hi Gecko,
If it is (as you think) a time independent median calculation ie year-by-year then that is a potential misrepresentation - remember the real world data are a valid realisation (the only realisation) of the physical process. If the computer simulated realisations cannot individually reproduce the flattening of temperatures for 20 years then the physical model is not valid.
These differences in presentation and description are very important when deciding whether the graphs are actually valid comparisons or not. Its very easy to mislead an uninformed audience in this way.
However, to be fair we don't currently know what actually is displayed, we are simply speculating. Where is that Richard Betts when you need him?
@ThinkingScientist
I note Smith et al.2007, final conclusion using the previous model finished with similar comments to those on the UKMO website about the next 5 years using hadgem3
However possible it is that there may be warming above the ensemble mean the point that many will be aware of is if predicitionmatches observations by 2017that we will be over 17 years without siginificant warming and that is also significantly cool(>95%) in the context of the AR4 predicitions for the 1995-2014
http://www.realclimate.org/index.php/archives/2008/05/what-the-ipcc-models-really-say/
(2nd diagram)
Are we going to see a publication of UKMOs new model or is that available somewhere already??
explaining the degree of internal (natural and unnatural) and external forcing for the different runs of this model producing the ensemble mean and or a longer term prediction??
@ThinkingScientist
Quite. But what exactly would represent a "median outcome" for the model over a period?
I think it is OK conceptually. As in, what does the model infer would be the central tendency for temperatures at any point in time and what does the model infer would be the potential range of outcomes.
That leaves you with a line and a shaded range.
I would like to thank Richards Betts for his replies to this thread.
err Richard, every post to a blog seems to be guaranteed to have at least one error.
MO have tried to clarify the graph fiasco.
A good example of the old maxim - when you are in a hole - stop digging.
http://www.thegwpf.org/met-office-manipulate-forecast-data/
The updated version.
I really wish commenters here would read the Smith et al. paper describing the model which Dr Betts commended, and its supplementary information. They answer many of the questions concerning the model and hindcasting. Smith wrote, "We use the term “hindcast” to refer to a forecast made retrospectively using only data that would have been available at the time." However, as Barry Woods points out, the retrospective periods shown on the MO page were apparently driven by historical forcings, as the Pinatubo period shows clearly. [The paper did not do this.] I would agree that calling them "previous predictions" is somewhat misleading.
I suspect that the reason for the 5-year prediction horizon, rather than the previous 10-years, relates to the point made in the paper that a bias appears in temperatures after running the model for a while: "This bias is essentially absent ... for lead times up to six years... However, [the model] does develop a warm bias in Ts beyond year six, rising to 0.07°C by year nine."
.
Smith, noting that the forecast run, initialised in 2005, shows no net warming through 2008, wrote that the model predicts "further warming during the coming decade, with the year 2014 predicted to be 0.30° ± 0.21°C [...] warmer than the observed value for 2004. Furthermore, at least half of the years after 2009 are predicted to be warmer than 1998, the warmest year currently on record." [Inclusive of the effect of removing their estimate of the bias.]
This is all part of what seems to be slow progress towards improving the quality of the climate models. The new model is better than the old; that's good. Yet it still shows a clear bias in predictions; not so good, but they're aware of it and are presumably beavering away at finding its cause. Smith tried to compensate for the bias, but it doesn't seem likely that 2014 will show a 0.30°C increase over 2004. Is this model bias, or is it due to some other effect such as solar? We don't know.
What *does* seem clear to me is the scientific baselessness of referring to model predictions of 2100, using them to inform public policy, when it is evident that models diverge from reality in less than a decade. By all means, let's allow Hawkins, Smith, et al. to continue their researches. But I'd like to see a greater admission from those involved as to the current unreliability of projections ten, twenty, or a hundred years in the future. As far as I can tell, MO still claim that they can reliably project distant future warming with a system which as Smith shows, displays an upward bias within a time horizon of five to ten years.
I guess plotting a curve in white over a basically white background, is another way of hiding the decline!
A lot of climatology graphs don't seem to be meant to be read!
A slightly less formal introduction to the limitations of models.
http://thepointman.wordpress.com/2011/01/21/the-seductiveness-of-models/
Pointman
HaroldW - Thanks for the links.
I have just read the paper and agree with HaroldW the astonishing thing about the last model is how far the model diverges from observations in only 5 years and embarassing for Smith et al.,
this is the obvious answer to why the metoffice have only presented 5 years,
as I asked above are they going to publish what they presented on the website
http://www.metoffice.gov.uk/research/news/decadal-forecasting
Richard,
"Thanks for all the comments and feedback though, it is very useful to be able to see what people think. Thanks for all your interest."
Does the Met Office run training courses in how to remain utterly pleasant when surrounded my an irritated mob? You're remarkably good at it.
My old Granny had symptoms like that, just before she went......
Seriously - hope you feel better soon - in the meantime, keep up the excellent work - although you probably don't feel much like it..!
Oh - and don't get too close to your computer - we don't want to catch it....
HaroldW
Thanks for that I read Smith a couple of years ago, time to have another look.
I think there are/were other issues with DePreSys maybe someone can make sense out of the following link? at present I have to admit I am having trouble comprehending the issues.
"Issues with anomaly assimilation in DePreSys:"
"Case study of the North Atlantic rapid warming in the mid 1990s"
Summary
"Analysing the ability of DePreSys to forecast the rapid warming of the North Atlantic in the mid 1990s highlighted that:
1. Many DePreSys hindcasts warm too early
2. DePreSys hindcasts are undergoing significant drifts
The problems seem to be caused by fundamental drawbacks of the anomaly assimilation method and highlight the need for more research into the initialisation of decadal forecasts"
http://www.met.reading.ac.uk/~swr06jir/presentations/JIR_Utrecht_poster2.pdf
Meto now suggedt they may review the misleading wording 'previous prediction'. See gwpf post.
Meto now suggedt they may review the misleading wording 'previous prediction'. See gwpf post.
Jan 11, 2013 at 3:10 PM | Unregistered CommenterPaul Matthews
Indeed Paul, the original caption shows how they intended to present the graph but the result would have shown a massive step. Hence they ran a hindcast with the new model to 'smooth the decline'.
BUT forgot to change the caption.
Richard has said here that the new model is 'expensive' to run but they clearly found the resources to run the smoothing hindcast.
Did they run on to 10 years and edit it back when they saw that massive dip at the end ?
Was one decline smoothed and another edited out ?
Am I alone in thinking that the Met Office would be justified in thinking that this is one hell of a stink being made over some apparently poor wording in a document, and that if this is the level of skepticism they have to put up with, they might reasonably assume that all skeptics are raving nutters?
We've all seen this story evolve and how the metoffice and the media handled it in reaction to blog pressure. Save for the fact that they were at least forced to report it, and were a little quicker to respond than after the Climategate leak, the coverage was predictable and lamentable The same attempts to ambiguate and downplay came from predictably the same CAGW promoting stables.
The metoffice attempted to sneak it out without comment on Christmas Eve, while instead a full press release story from a conjured up rising extreme rainfall trend (out of a pile of pure statistical trendlessness), was given full publicity, playing on the recent wet weather in a naked bit of 'messaging'.
The BBC and Guardian larded their stories with misleading statements and added the carefully selected quotes conveniently pre-supplied in a briefing note by Fiona Fox's politically hard left pressure group the Science Media Centre (you have to admire her efficiency), as David Whitehouse reveals in his post-mortem of the whole affair
http://www.thegwpf.org/met-office-warming-2017-media-do/
In an environment where Louise Gray telegraph environment reporter thought, that the new forecast meant, by 2017 there would be an EXTRA 0.43c warming from now!
The problem lies elsewhere than the sceptics
Ie she thought met office projection meant, 0.8C from 1850 plus an extra 0.43C by 2017
As did very many members of public based on the BBC article as it was originally written, see leo hickman's guardian article comments, where he led with the original misleading bbc article.
Ignorance is high amogst some believers, despite being pointedcto the actual met iffuce press release and new forecast, that contradicted this mistaken belief..
Ie not an extra 0.43c by 2017, the projection sayin about the same as now by 2017
The magnitude if this misunderstanding is large, as0.43C by 2017, would be more warming in next 5 yrs, than the wholes if the 90's and maybe the 80's as well
HaroldW:
Head of nail, meet hammer.
Am I alone in thinking that the Met Office would be justified in thinking that this is one hell of a stink being made over some apparently poor wording in a document, and that if this is the level of skepticism they have to put up with,
they might reasonably assume that all skeptics are raving nutters?
Jan 11, 2013 at 3:35 PM | Unregistered Commentersteveta
=============================================================
Are you alone ? Yes I think you possibly are. If you cannot see what has happened here.
To produce a graph's detailed lengthy caption only to find that resulting graph looked stupid, then to insert an unrelated graph from the new model they had NOT intended using in the first place while forgetting to cover up their original intention with a revised caption is rather more than revealing don't ya think ?
They've reluctantly and hurriedly smoothed the decline by hindcasting hoping no one would notice. And Richard did not think they had resources to do a decadal run - well they did -
5 years forward - 5 years backward.
Did they actually do the forward decadal run too editing it when they saw where it was going ?
Perhaps the community chest will send the metoffice metorphorically to jail and we can advance back to go with the temp-CO2 chicken and the egg, the 7/10ths ocean and its heat storage capacity, and the ice core 800 year lag record.
steveta,
"Am I alone in thinking that the Met Office would be justified in thinking that this is one hell of a stink being made over some apparently poor wording in a document, and that if this is the level of skepticism they have to put up with, they might reasonably assume that all skeptics are raving nutters?"
Maybe I could explain why it irritates me, and why I think it's important. By claiming that hindcasts are "previous predictions", while simultaneously removing their genuine previous predictions from public view, they are giving the casual observer the impression that they are much better at making climate predictions than they actually are. Which then gives the impression that they have an accurate understanding of the climate system. Which implies that we should listen to their dire warnings. Which implies that it's a good idea for the government to ask me to pay for windmills, while I wonder how to pay the heating bill.
In my post of 11.18am I actually tried to defend the Met Office, suggesting just a case of sloppy wording. Now, I read the following (from GWPF) that someone from the MO has actually written "...our new model HadGEM3′s hindcast much more closely represents real-world observations – indicating a higher level of skill."
I give up - now I can only agree with Martin A - close them down.
But before they go Prof Slingo must write to the prime minister, to inform him that all of their attempts to construct a model to predict future climatic conditions have failed, and that there is no confidence that such work - now being described by the MO as experimental - can ever produce such a model.
It is then up to the PM, and his government, to determine if the current policies that are freezing our old to death, and sending our children to bed hungry, should be changed.
steveta 3:35pm - if they were so far up their own importance - well yes that is possible.
BUT remember these people are our servants not the other way round. We are bankrupting our country based on their advice. This is not just a cosy discussion about science. It is about people's lifes.
As Richard Feynman told us - "the easiest person a scientist can fool is him/herself"
We are grateful to Richard Betts for interacting here. We are just seeking honesty.
James, Roger, yes, exactly. There is no excuse for the press release or the stupid responsed from Dave Britton and other met office people.
Post-climategate they should know they have to be ultra-careful and squeaky clean.
That would be some day that saw such a letter, Roger Longstaff! Flights of pigs would do fly-pasts out of sheer joy.
I suspect 'close them down' would follow if the National Audit Office looked at the costs incurred and harms inflicted by the climate prognostication section of the Met Office, and compared that performance with the pretensions by which they seem so readily to win more funding from the unwitting taxpayer.
First we pay them for delusions and distractions. Then we pay again when they are taken seriously. And who knows what the final reckoning will be for panic-driven polices such as the Climate Change Act which emerged from a political culture deranged in part by the Met Office.
I have only one thing to add to this debate, and it comes from the IPCC:
1. ” … In climate research and modelling, we should recognise that we are dealing with a coupled non-linear chaotic system, and therefore that the long-term prediction of future climate states is not possible.” IPCC
3rd IPCC TAR, Section 14.2 “The Climate System”, page 774.
And here we have the good old Met Office toiling away at the impossible, moreover, we are discussing the possiblilities of something that's impossible being correct.
I've said it before, but will repeat it, the current hiatus in the warming will eventually dawn on the politicos, which is why Julia Slingo and her fellow activists in the Met Office have been declaring years as the "nth warming on record", rather than pointing out the difference from year to year for 15 years or so have been close to zero. Eventually someone will get the message through to the politicians that despite an 8% increase in CO2 over this period the temperatures have remained stable. Handwaving "natural causes" can only go on for so long before someone asks why natural causes can cause cooling and not warming, to which, of course, there is no answer. Nor do they know what the natural causes are that have stopped the rise in temperature with a concomitant 8% rise in CO2. So, admitting that the warming has stopped and pretending (see IPCC statement above) that you know when it will start again has given them breathing space. It's not science, it's political activism, much as I admire Richard Betts, Tamsin and Ed Hawkins and the many other scientists beavering away at trying to understand the climate, the scientific establishment in this country has been taken over by political activists. They will, eventually, pay the price, and the innocents with them unfortunately.
roger harrabin @RHarrabin
Insider at @metoffice tells me scientist who put out decade forecast on 24 Dec was meeting an end-of-year deadline before starting hols.
The way the MO handled the recent published material is not the main cause of "all the stink". Taxpayers and scientifically minded people are just totally dismayed by the way the MO has abused its position over many years by becoming part of the alarmist movement.
Instead of producing objective statements about the science we get endless alarmist threats that do not stand up to scientific scrutiny or the subsequent reality of observations. The policy makers and management within the MO have done huge damage to this once respected organisation.
.
all computer models are experimental. Lord this is just more pea under the thimble nonsense. It is not possible to predict the unpredictable. Someone up thread very kindle pointed out that even the numpties in the IPCC acknowledged this. Even the climate doesn't know what it is going to be doing in 5 years let alone 80 or 90.
Here is a prediction, by the year 2100 climate scientists will have openly admitted that their computer models have not been able to predict any future climate with a degree of accuracy that actually had any value.
Dolphinhead: "by the year 2100 climate scientists will have openly admitted that their computer models have not been able to predict any future climate with a degree of accuracy that actually had any value."
Well, science is wonderful so I wouldn't necessarily agree with your prediction. But I'm pretty sure they'll say, "the computer models of 2010 were unable to predict any future climate with a degree of accuracy that actually had any value."
Good heavens, look at this other tweet from Harribin:
One can debate the contextual denotations and connotations of 'innocent' and 'naive' and 'bloody stupid' ..... Alarmists will try to say it's about wrongly giving 'ammunition' to evilll 'deniers' but more impartial observers will say 'look, someone inadvertently revealed ....'
BTW, after literally minutes of cutting-edge work, I can finally release the output of my climate model. My model uses experimental cut-and-paste methodologies. The output is in the form of a graph. Global temps are shown in red. In blue I have spliced together a 42 year hindcast and a 60 year forecast.
http://postimage.org/image/e8wvgak8r/
Focus upon what Harribin's MetOffice 'insider' might mean by such remarks......
.... Then think about those words in relation to all kinds of public utterances by various scientists and public figures through the years..... How much of the output from Team Alarm has focused upon 'stay on message' rather than doing the fullest, most objective effort possible across all relevant topics of science, statistics, and public policy....