Bishop Hill Met Office or bookie's office?
Mar 29, 2013 Climate: MetOffice Climate: Ward Roger Harrabin takes a wry look at the Met Office's three-month forecast for last spring. These longer-term forecasts are not published, so Harrabin got it under FOI. The results are most amusing:
The Met Office three-monthly outlook at the end of March stated: "The forecast for average UK rainfall slightly favours drier than average conditions for April-May-June, and slightly favours April being the driest of the three months."
A soul-searching Met Office analysis later confessed: "Given that April was the wettest since detailed records began in 1910 and the April-May-June quarter was also the wettest, this advice was not helpful."
The whole piece is worth a look, but I was struck by this bit:
The Met Office explained it this way: "The probabilistic forecast can be considered as somewhat like a form guide for a horse race.
"It provides an insight into which outcomes are most likely, although in some cases there is a broad spread of outcomes, analogous to a race in which there is no strong favourite. Just as any of the horses in the race could win the race, any of the outcomes could occur, but some are more likely than others."
The analogy between weather forecasting and backing horses is amusing, particularly in the light of a Twitter conversation between the FT's Jim Pickard and Bob Ward last week:
HOTTEST MARCH EVER AS MINI-HEATWAVE HITS UK: Is this the worst weather forecast since Michael Fish?
.@PickardJE Yes, that's what happens when journalists confuse Ladbrokes with the Met Office.
Perhaps it's different when the Met Office confuses the Met Office with Ladbrokes.
Reader Comments (99)
On Unthreaded, noting Professor Slingo's reference to "the difficulty of constructing long-distance forecasts, given the UK's position at the far edge of dominant world weather systems", I suggested that she's trying to tell us that, although the Met Office may get its longer-term UK forecasts wrong, that in no way impugns the accuracy of its global forecasts.
"This advice was not helpful."
The epitaph of climatology.
Could the Met become self funding if Ladbrokes accepted bets on weather forecasts ?
Gadfly.
Slingo on R4Today admitted today they had used the "precautionary principle". So much good use for all those supercomputers, uh.
What is really strange is that they would have done better by use of a pin. I remember one of their long range forecasts last year whihc read something like " 30% dry; 30% wet, 30% averagë".
They are useless at long range forecasts because their models are bad. Why the hell don't they shut down their climate unit and concentrate on what they do averagely, the daily forecasts.
Here in New Zealand we have a web site http://ipredict.co.nz which allows one to stake real (though small) amounts of money on the outcome of various economic and political and other propositions. Current examples relevant to this blog include:
http://www.ipredict.co.nz/app.php?do=contract_detail&contract=IPCC5.SEARISE.GR55
Recently closed ones included for the global temperature (according to GISTEMP) for 2012 to be higher than 2011, or a record:
http://www.ipredict.co.nz/app.php?do=contract_detail&contract=TEMP.2012
http://www.ipredict.co.nz/app.php?do=contract_detail&contract=TEMP.2012.HIGH
I don't know whether there is such a site in the UK.
It would be interesting to see how such a site performs compared to the Met office or climate scientists. They tend to do quite well on predicting political and financial events.
A sign that the tide of political madness over CO2 and climate has turned will be when the Audit Commission is directed to assess the losses to the UK brought about by Met Office computer models used for anything other than short-term weather forecasts. I am thinking of fall-out from Chernobyl, volcanic dust from Iceland, and of course their various speculations on climate months ahead, and for as many years ahead as we now have verification evidence to assess. I know the Met Office will claim net benefit. I know I am not prepared to take them at their word, nor rely on their data analysis alone, any more than I would, say, rely on Labour Party analysts' assessments of the impact of the last Labour government on the wellbeing and prospects of the UK.
Anybody else catch the "phrase of fashion" otherwise known as...
"not helpful"
I believe it was the Muir-Russel pannel that resorted to that phrase as their strongest-possible-term...
As I woke up this morning I could hear Slingo being interviewed on Toady. I was struck by the off-hand way she plucked the 'about 65%' probabilistic accuracy for forecasting from the air and the comparison with her organisation's rectitudenal obsession with the claimed accuracy and 95% confidence of their global climate average temps for 2060! Makes one wonder what kind of discussions she has with Richard Betts...
The 'precautionary principle' in weather forecasting is a cop-out to help forecasters sleep at night. Basically, you take the worst-case option and publish that on the assumption that it is better to forecast howling gales and get a breeze, than the other way round. This amounts, in my view, to unprofessional behaviour. They should be using their best judgements and sharing them with us. We the user can make the 'worst case' decisions ourselves using our special knowledge of what is at stake for us. We do not necessarily benefit from others doing that for us.
"not helpful" = "wrong" or "the opposite of what happened". You chose.
And yet the same people using similar models have no issue with making claims about what WILL happen many years ahead even claiming accuracy to two decimal places .
So that's why they hire people with names like Richard Betts...
You're a genius Josh!
It all makes perfect sense to me. When the Met Office makes a statement, the opposite is true. So the earth is cooling, their models don't work and they are pretty useless at forecasting.
Now I understand what they told Beddington.
It was easy to predict that June 2012 would be wet. It so often rains on the queen, and that is not intended as a pun.
"She says last year's calculations were not actually wrong because they were probabilistic. "
So, using Slingo logic, the statement "The Met Office is probably a criminal waste of money" is not wrong.
I am trying to think what made me angrier: the school girl giggles of Madames Montague and Slingo or the complacency with which the head of this multi-mega-million pound user of IT said that they were "helpful" 65% of the time !
So they can't even be mildly accurate 2/3rds of the time ! I bet a pine cone could achieve better results.
(The average probability would be 20%). - from the BBC report
So what I understand Harrabin to be saying is that there was a 20% probability of the period being wettest OR driest. Covers all bases I suppose, but it suggests a complete lack of intelligent thought.
When it comes to medium or long term Met Office forecasting I do not think I have seen anything other than a forecast for average conditions. It is a safe bet that average conditions will prevail. This tactic is more to do with protecting MO accuracy statistics than providing a proper weather service. What is the point of a forecasting agency if they never tell us when abnormal conditions are due?
It seems to me that the accuracy of short term forecasting is good but not any better than it was 40 years ago. So why are we paying a fortune for a massive, modern MO that is not publically prepared to stick its neck out and warn the public and business about unusual weather patterns 3 months out?
The real problem is that they rely on obviously faulty climate modelling to predict weather further out. The sooner they ditch the dogma the quicker we will get back to decent forecasting.
Sligo claimed that they had made several forecasts for the period in question. Was that to try to get at least one correct? Just like a broad bet on roulette- same outcome- WRONG.
per Josh or why all Betts are off nowadays
I wonder how this year's April, May and June will pan out?
"Met Office 3-month outlook"
"Period: April 2013 - June 2013 Issue date: 21 March 2013"
SUMMARY - TEMPERATURE:
"For April below-average UK-mean temperatures are more likely than above-average. For April-May-June as a whole above-average temperatures are weakly favoured. However, there is still a significant chance that this period will be colder than it was in the majority of the last 10 years.
The probability that the UK-mean temperature for April-May-June will be in the coldest of our five categories is around 15% and the probability that it will be in the warmest category is around 20% (the probability for each of these categories is 20%)."
http://www.metoffice.gov.uk/media/pdf/s/2/A3-plots-temp-AMJ.pdf
SUMMARY - PRECIPITATION:
For both April and April-May-June as a whole the uncertainty is large, leaving the forecast largely indistinguishable from climatology.
The probability that UK precipitation will fall into the driest of our five categories is around 20% and the probability that it will fall into the wettest category is also around 20% (the probability for each of these categories is 20%).
http://www.metoffice.gov.uk/media/pdf/s/r/A3-plots-precip-AMJ.pdf
"Contingency planners"
http://www.metoffice.gov.uk/publicsector/contingency-planners
Met Office "or both April and April-May-June as a whole the uncertainty is large, leaving the forecast largely indistinguishable from climatology."
_____________________________________________________________________________
So what does that mean for climatology? Uncertainty large apparently.
They can't even get their simplistic analogy correct. It would be the equivalent of the horse racing ODDS not the form guide (which is part of the raw material for determining the odds).
A 20,20,20,20,20 forecast is exactly the same as saying WE DON'T KNOW! Which is fine, thanks for admitting it, leave your pass on the way out. It would save us a fortune.
Even if a historical analysis showed that the odds really were 20,20,20,20,20 then the forecasts would still be of no use at all. It would just indicate that forecasting is a waste of time in this scenario.
For a useful display of skill I would expect to see forecast spreads like 15, 65, 15, 4, 1 AND for this to match within reason what actually happens over time. Without that sort of level of skill, as I have said before, it is just (very expensive) "astrology for intellectuals".
The figures must exist to show how good these guys are. It's way past time to call them to account.
What ever you think of Piers Corbyn he is way ahead of the Met with only his science and meger facilities. He also has a good take on CO2 and Climate !
Whenever I read this tosh from people in the Met Office I picture Sir Alan Sugar listening to their feeble explanations across the table and I imagine his response. It would make good television.
Perhaps they should get Jeremy Paxman to do the long range forecasts as well. http://www.youtube.com/watch?v=HMAt8ZXqtbc
Its worth remember way the MET stopped given their mid and long range forecast to the public in the first place. Because they got it so wrong so often it became a running joke . And by 'lucky chance ' these mistakes where given results that favoured 'the cause'. Some may say that that merely reflected the bias they built into their model shas the MET allowed itself to be come a 'political' not science driven organisation.
I long ago gave up on the MET for providing a short term forecast with any reliable degree of accuracy.
I have been using http://www.meteociel.fr/modeles/gfse_cartes.php?&ech=180&mode=1 for over a year now and find their graphic forecasts out to seven days regularly conflict with the childish output on BBCTV weather yet are correct many times more often than not.
An immediate example is the MET office last weekend forecasting the possibility of heavy snow over Easter spreading in from the SW whilst Metiociel showed an unwavering blocking high throughout this weekend.
I confidently maintained my plan to drive the length of England to visit siblings, booked Odysseus into cattery and have been vindicated by the outcome.
Vive la France!
Snotrocket said:
" I was struck by the off-hand way she plucked the 'about 65%' probabilistic accuracy"
This is what struck me too, the £billions poured into the Met Office provide a _15%_ improvement over the toss of a coin.
At least it's money well spent.
>:-|
Nial
Wonder if their useless forecasting has anything to do with the Weather Generator Project mentioned in the Climategate mails.
Seems it was more important to "tell a story" than ensure accurate forecasting.
“I can’t overstate the HUGE amount of political interest in the project as a message that the Government can give on climate change to help them tell their story. They want the story to be a very strong one and don’t want to be made to look foolish.”
http://foia2011.org/index.php?id=2445
Interesting how many of their calibration sites were based at airports.
http://ukclimateprojections.defra.gov.uk/media.jsp?mediaid=87941&filetype=pdf
I have today written to Nick Clegg with links to the Economist article and the Cold on Nelson's Column bit and added the following
"The only way to create wealth is to do things more efficiently. Making energy artificially expensive destroys wealth - except if you are a Yeo a Deben or a PM's father-in-law - so the policies you have promoted and which are still being mis-sold by your colleague Ed Davey are achieving exactly what they are designed to do; they are causing actual harm to the citizens of the country now and serious damage to the economy. If you are naive enough to think they will have any effect on the amount of CO2 in the atmosphere then you trully are deluded. There are currently plans to build some 1200 coal fired power stations world wide. Leaders of other countries are doing what their citizens need and providing cheap energy.
And lest you are in any doubt there is no, I repeat no, empirical evidence to suggest the small amount of CO2 produced by human beings is in any way harmful. All the scare stories you have been fed are based on the output of computer models and sold to you as 'scientific evidence'. You have told me on a number of occasions that we have to agree to disagree. That is easy for you to say where you are in a position to continue to pursue policies inimical to the interests of the citizens of this country. For my part I will continue to attempt to help you see what a terrible mistake you are making.
The policies you are promoting can best be described as 'not helpful' - for enlightenment see - http://www.bbc.co.uk/news/science-environment-21967190
Still you will be able to tell your grandchildren that you were one of the architects of Britain's downfall to third world status and that you were responsible for causing death and suffering to thousands of the most vulnerable members of society.
It is my greatest wish and hope that people like you will be called to account for the massive harm you have done and the reason I will continue to communicate with you is so that you cannot say you did not know.
Time to open your eyes and see what you have done"
On past performance this will elicit a non-response. We will see. The tide is turning and it will be increasingly more difficult for our elected leaders to continue to pursue policies that are 'not helpful'.
Surely it's not the job of the MET Office to issue forecasts with the precautionary principle applied? That means they see themselves responsible not just for weather forecasting (and they're apparently not very good at that even after years of doing it), but also to determine what weather is best for us. I'd like to just have the weather forecast, please, and then I'll determine whatever precautions I see fit. What suits the people at the MET Office as 'best' weather may not be the same for everyone.
And I find their three months forecasts almost useless in terms of information - 'there'll be a 20 % probability it'll be colder than usual, 30 % it'll be warmer' . . . I can do as well as that. Here you go: June will be warmer than April - 75 % confidence level!
Surely it's not the job of the MET Office to issue forecasts with the precautionary principle applied? That means they see themselves responsible not just for weather forecasting (and they're apparently not very good at that even after years of doing it), but also to determine what weather is best for us. I'd like to just have the weather forecast, please, and then I'll determine whatever precautions I see fit. What suits the people at the MET Office as 'best' weather may not be the same for everyone.
And I find their three months forecasts almost useless in terms of information - 'there'll be a 20 % probability it'll be colder than usual, 30 % it'll be warmer' . . . I can do as well as that. Here you go: June will be warmer than April - 75 % confidence level!
But according to their annual reports and accounts they're doing an excellent job!!!
"2010: AN EVENTFUL 12 MONTHS
With a challenging backdrop of natural events, and increasing
economic and operational pressures, the Met Office delivered
effectively against its Business Performance Measures (BPMs).
Not only did we meet weather forecasting BPMs — for
example, accurately predicting extremely cold conditions in
December 2010 — we also did well reducing carbon dioxide
(CO2) emissions and developing other areas of sustainability
across the Met Office.
The challenges of 2010 were many and varied — beginning in
April with the eruption of the Eyjafjallajökull volcano in Iceland
and its unprecedented impact on international air travel. As
one of nine Volcanic Ash Advisory Centres worldwide, we
worked closely with the Civil Aviation Authority and National
Air Traffic Services in the UK, as well as airports, airlines and
other national weather services to inform decision-making
through our understanding of the ash cloud’s movements."
http://www.metoffice.gov.uk/media/pdf/b/q/ara11amend.pdf
As many will know this was an unmitigated disaster with unnecessary closures causing huge losses to our aviation industry and all because computer models had been preferred to observational data
http://www.eureferendum.com/blogview.aspx?blogno=70960
http://www.eureferendum.com/blogview.aspx?blogno=71163
And yet - AND YET - they can confidently predict the climate in a hundred years time..!
Oh - sorry - that's 'climate' - not 'weather' - not the same thing at all...
What is it they call their £60m super-computer - Deep Black..?
Should be called Deep..... oh, I see you're way ahead of me...
Oh my. Roger Harrabin of the BBC actually FOIAing for evidence? He should be careful or he might end up becoming a science reporter.
The Met Office is not the only weather forecaster that sometimes gets cold feet and takes their art back underground. Last week the Wall St Journal reported that the world's most notorious weather forecaster, Punxsutawney Phil, was facing prosecution:
http://blogs.wsj.com/law/2013/03/22/prosecutor-chucks-book-at-groundhog-for-bad-prediction/
Lucia has details of the charge-sheet against the warmist groundhog: http://rankexploits.com/musings/
In "The Signal and the Noise," Nate Silver documents how weather forecasters tend to have a "wet bias." If the unforecasted rain ruins the picnic, weather people receive more criticism than if it is an unexpectedly nice day.
One of his sections is titled: "How to know if your forecasts are all wet." :)
Do forecasters in Britain have a "barbeque summer" bias, or are those predictions just made by people who are "not helpful"?
According to Julia Slingo in her testimony to the Science and Technology Committee -
Professor Slingo -
".....At least for the UK the codes that underpin our climate change projections are the same codes that we use to make our daily weather forecasts, so we test those codes twice a day for robustness. "
"Q210 Graham Stringer: You do not always get it right though, do you? "
"Professor Slingo: No, but that is not an error in the code; that is to do with the nature of the chaotic system that we are trying to forecast. ...."
http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/387b/38724.htm
Credit to Harrabin for doing SOME journalism and getting the MetOffice forecast under FOI (if that's what in fact he did).
However, the way he teed up the Slingo interview to simply give her a platform to "explain" how everything was OK, was pathetic.
For instance, he assured us that the forecast wasn't "wrong" because it was a probablistic forecast. Eh? Sorry?? How does that make a forecast for April, favouring drier-than-average conditions, issued ON THE LAST DAY OF MARCH, "not wrong"?????????? Let's consider the converse: was the forecast correct? Plainily not, hence, it was WRONG.
Why did neither Harrabin nor Montague ask Slingo what the MO's forecast was of April NOT being in the wettest category was (i.e. the sum of all the possible outcomes other than the wettest)? This, presumably, would have been something like an 85% probability, accoring to the MO, that April would not fall into the wettest category. And yet it did. How no earth is this "not wrong"?? On this basis, the MO can argue that its forecasts of CAGW at 90 - 95% certainty or whatever are also "not wrong" when they eventually don't pan out.
What the hell is our state broadcaster, which is supposed to be there, I hear them say, to speak truth to power, doing letting them get away with this crap??
As for her fact-free claim that they get it right 65% of the time, I'd LOVE to see the facts to back that up. I seem to recall Paul Hudson (apparently the only person employed by the BBC who remotely knows what he's talking about on these issues) point out that the MO's annual temperature anomaly forecasts (now abandoned on account of being irredeemably rubbish) were wrong something like 10 times out of 11 on the HIGH SIDE. Warm bias much? I wonder how that could have arisen??
She did, of course, refer to the entirely unsurprising fact that the MO's forecasts improved the closer they got in time to the impending weather (Really? No shit, Sherlock!). Perhaps her 65% accuracy figure is for forecasts issued the night before? Colour me unimpressed.
From Harrabin's transcription of Slingo:
//
"When asked about weather predictions in the coming months Prof Slingo said the cold weather could continue into the middle of April.
She added: "Our monthly forecast favours cold conditions continuing.
"Into the summer it's much more difficult to predict but we're expecting a return to near normal conditions into May and then June but of course its important to emphasise that this is only one of a whole sequence of forecasts we give."
//
I guess his editor must have cut Roger's "FFS - ROTFLMAO" off the end the piece...
When I last lived in the UK, I used to ride a motorbike to work (Suzuki GS850 even though you weren't asking). If it was a fine morning, before leaving the house I used to listen to the BBC radio weather forecast. Then I knew whether to take my wet gear or not. In two years, coming home I got wet 2-3 times. I thought that was a pretty good result.
Last year I made one of my thankfully rare visits to the UK. The BBC TV weather forecast was terrible childish stuff. A useless map with kiddy icons. No sign at all (unless I missed it) of a synoptic chart, which I'm led to believe is too technical for the kiddies. And no chance at all for me to make an informed decision.
What's happened to "inform as well as entertain"? NHK (modelled on Reithian BBC) weather always gives the synoptic chart as well as the fluffy clouds. NHK also uses maps and diagrams in mainstream mass entertainment programmes. Viewers here are invited and expected to lift their game rather than NHK lowering theirs. There are reasons why Asia is moving forward wrt to the western nations.
Sorry, sorry [/rant]
Makes one wonder what kind of discussions she has with Richard Betts...
Mar 29, 2013 at 9:43 AM | Unregistered CommenterSnotrocket
The same. Betts needs to tell her that the models are great or he will be unemployed. His role is climate impacts. No model, No job.
Mar 29, 2013 at 2:21 PM | Registered CommenterHector Pascal
The BBC changed to their New Zealand bought software a few years ago amid the most complaints ever seen on the BBC. Needless to say the arrogant little s&é"ts just ignored them.
Mar 29, 2013 at 2:21 PM | Registered CommenterHector Pascal
Hector, when I lived in the UK I too rode a Mobike (honda 250). I remember two particular periods. One was a June (mid 80s I think) when it rained everyday. You tend to remember the rain on a bike. The other was the winter. We had 45cms of lying snow which froze hard. I couldn't get the bike to cross the ruts of ice.
Hi Hector - my cat sends his regards, mewing "long time no see".
Ah! Those synoptic charts that allowed personal selection of a good fly fishing day on lakes up to 50 miles away. Almost never got it wrong in 20 odd years - very important when hiring a boat at some expense.
Now it's barely fit for townies leaving their high rise to visit the local takeaway.
Met Office News Release - April 1st 2014.
We now regret saying in December 2012 that on balance March 2013 would be more likely to be very mild than very cold. - this advice was probably not helpful, although it is likely (at 95% level) that nobody believed us anyway..
http://www.thegwpf.org/met-office-forecast-march-above-average-uk-mean-temperatures/
The GWPF says the Met Office’s temperature forecasts issued in 12 out of the last 13 years have been too warm. Well worth going through.
http://www.thegwpf.org/warm-bias-met-offices-disastrous-track-record/
That means the Met office might have got it right in 1 out of 13 years (7.5%). Back to the dartboard, the dice or the coin.
Trouble is, people, you just wait until the MO get a forecast wrong (inevitable since they are probabilistic), then weigh in as if all of their long-range forecasts are wrong!
Quite a good demonstration of how most people fail to understand the notions of risk and possibility
In North America, weather forecasting has become entertainment in that it has to compete for viewers and on most news bulletins they use a 'stinger' before the commercial breaks about upcoming weather reports so you don't change channels (to avoid the ads). Consequently, there is an obvious (and somewhat understandable) tendency to dramatize whatever weather they are promising/reporting on. Furthermore, there is so much forecasting (and almost no reporting on actuals) that the viewers don't ever know how accurate the forecast was and are left thinking that the actual temp was what the forecast promised. Consequently, impressions of rising temperatures and extreme weather are not based on the actual weather but on the forecast itself.
To get back to the MO. How much do you think people would have noticed if they had not been spectacularly wrong and only been just a bit on the warm side? People would have simply believed that it was a warmer/drier than usual summer because of the forecast and not based on the actual numbers. The warm bias in MO models thus becomes a self-fulfilling prophecy in the minds of the public (and probably the minds of the forecasters as well). This is why there is a sudden realization of no warming for 16 years - because everyone was believing the forecast until someone in the MSM finally had the courage to publish the actual data.
The question then becomes why does the MO do this when they have no need to spice up their forecasts to keep viewers tuned in over the adverts? The precautionary approach (not principle in this case) that it is better to warn of bad weather so that people are prepared does not apply in the case of warmer summers in the UK since a "barbeque summer" would be just great. It must be ideological and that really has no place in a publicly funded body.