Monday
Jan312011
by 
Bishop Hill
Bishop Hill Everyone's a winner
Jan 31, 2011 Climate: MetOffice This comment just appeared on the Met Office thread, courtesy of Thinking Scientist. It's too good not to have a post of its own:
I looked at the documents Katabasis got from the FOI of the MET office. The predictions from the Met are even poorer quality than appears at first glance because their categories for mild average and cold overlap!
Mild -0.1 to +1.3 Probability 30%
Average -0.5 to +0.6 Probability 30%
Cold -1.5 to +0.4 Probability 40%That also means their probabilities make no sense, and gives them a double dip, or even a triple dip! If the actual anomaly was, say, 0.0 then it would be in all three categories. Brilliant! Everyone's a winner...
Can anyone think of a rational explanation?
Reader Comments (56)
As you so rightly point out, the overlaps mean their probabilities do not add up to 1, which is the only thing they got right as the probability of an event outside their range was clearly non-zero (it happened). As the Met Office themselves would say, probabilistic forecasting should be kept away from those too stupid to understand it, ie themselves.
Seems a bit of a waste of money buying a £33m supercomputer for people who can't get their probabilities to add up to 1, though. But that's what you get when you hire activists instead of scientists to run it.
The commenter name: ThinkingScientist is a propos. Does the MET Office have a training budget to offer remedial lessons in math and stats to it's own?
Silly you, expecting anything rational from the Met Office.
Post-normal arithmetic.
Good grief. I do not see how there can be any coherent explantion for these predictions - except multiple parallel universes.
The categories mild,average and cool are for the whole of Northern Europe. And their predictions are for N Europe rather than the UK. So, I think, a winter can only end up in one of the categories, based on the outcome of temperatures for all of europe. The temperature in the UK might then fall within the guide-line temperatures. But there is something like a 1 in 5 chance that the UK temperature will be outside of the range. (Eyeballing the graphs in the pdf file of the FOI stuff).
If you worked in the Cabinet Office and you received this, would you not simply scribble on it - "George, I think I've found a way for you to save some money" and stick it in the internal mail for No11.
i think the met are comparing northern europe temps to uk temps here. certainly TS is looking at anomoly figures for uk average and not temps. i don't think TS understood the chart but it does seem a bit arbitrary at first sight.
I believe that the Met Office team are using their supercomputer as a giant abacus.
Can anyone think of a rational explanation?
Yes. They predicted winter.
yeh: the forecast is for northern europe. the table TS refers to is a comparison for UK temps with respect to northern europe temps, but not part of the forecast in itself. however in the text they state they disregard 'atypical' years (in regard NE - UK comparison), so they seem to be deriving anomolies of anomolies whilst rejecting data that's too anomolous.
Rational? No, not rational...
Harrabin was trying to explain this on the R4 news this evening. He even repeated the MO wonderful statement about the (low) probability of a "cold and wintry start to the season", which is another way of suggesting that winter might be wintry. My cat knows that.
You see? This is why Julia needs her shiny new supercomputer...
Maybe their supercommuter stopped and they couldn't afford a shilling for the Leccy?
Peter Walsh
Well, its nice to make a headline on Bishophill!
Re: Mark, my reading of the table is that the figures given are the UK temps and anomalies for the categories used for Northern Europe predictions. The categories used seem to be the Met Office prediction categories as stated, and with the probabilties given.
Re: Kevin's comment "Does the MET Office have a training budget to offer remedial lessons in math and stats to it's own?". Well, oddly enough I actually teach statistics and geostatistics training courses both to industry and to academia (at Imperial College, London). If they want my day rates, I am happy to give them a quote. Could even talk to them about my favourite topic of stationarity in auto-correlated time series, if they want. I even use global warming temperatures as the example...
Sure. This is possible.
To keep it simple let us assume that we only have probability masses (Dirac delta functions for the mathematical literati) at the temperatures -1.5, -0.5,-0.1,0,0.4,1.3 as this is the set of all of the temperature range end points. We also assume that these ranges include the end points.
Then we need to find the probability mass to assign to each of these temperatures to recover the probabilities above. Note that the probabilities do not need to add up to 100% as there will probably be some probability mass outside the range [-1.5,1.3]
This is a simple linear system. It has 7 unknowns with 3 linear equations and so has an infinite number of solutions. However many of these imply negative probabilities. If we constrain the probabilities to be positive then the number of solutions is smaller. Here is one solution
Temp Prob(Temp)
-1.5 12%
-0.5 4%
-0.1 8%
0 8%
0.4 8%
0.6 2%
1.3 4%
The total probabilities add up to 46%. So they imply a 54% probability of being outside the range. We can't say more than that. It also suggests that they had an 12% probability on lowest temp. The shape of the distribution is quite multimodal - i.e. more than one peak. And who knows what was going on outside this range.
In reality they surely had a continuous distribution. But given the information above, this is the closest we can get to guessing what sort of form the probability distribution of predicted temperatures took.
I'll get my anorak.
sorry if i miss understood TS but i wanna go watch some bbc propaganda now. but the probabilities are not given in this table and the categories given in table are for NE. not UK. it just says UK weather is likely to be like NE weather (but not very well quantified)
Re: Fred, the problem is solvable as you say and clearly there are possibilities outside the range (ie continuous PDF - after it happened!) but their prediction is quite explicit. They give three categories and the probabilities sum to 1.
Monckton's on now, so guess everyone will be off line for an hour or so!
I'm not a real mathematician, but how is an anomaly of +0.4 an expression of extra cold?
ThinkingScientist
Hahaha...good stuff.
Oh, did you want the unadjusted temperature ranges ?
I wonder how Mystic Met will react to being rumbled? Ignore it and hope it will go away??
My theory is that they selected the range boundaries (which look very arbitrary) to get the probabilities to add to 100% since they imagined that this is one of the things that mathematically illiterate readers of the report will ask - after all they are mostly politicians who did PPE.
They did this to avoid a phone call from Number 10 saying "Dave and Nick wanted to know why the probabilities only add up to 96%. They think you made a mistake. Can you correct and resend".
We could be dealing with confusion on overlaping probabilities. A simple analysis gives the results.
0.6 to 1.3 : 15%
0.4 to 0.6 : 10%
-0.1 to 0.4 : 35%
-0.5 to -0.1 : 20%
-1.5 to -0.5 : 20%
It still doesn't add any clarity to the Met Office forecast.
Hi Fred, probably, but its makes them scientifically illiterate. We can't judge the final outcome yet because this is a Dec/Jan/Feb prediction, and we have another month to go, but I think think we are way outside their limits. They gave a prediction based on a bounded PDF and got it wrong (again!).
The other thought I have comes from reading the FOI material - they state the historical chance of a cold winter is 1 in 5 but they then say this figure needs updating for the climate change trend and arrive at 1 in 20 for a cold winter in the current period. But there appears to be no attempt to check this against the historical data, instead it is a model based prediction.
I am going to have a look at the probabilities suggested by the MET office to see how they stack up against the actual data and statistical probabilities.
Probably after the furore that will follow the Monkton assassination on the BBC that is currently on as I write this.
Mac your probabilities give
[-1.5,+0.4] = 20% + 20% + 35% = 75%
[-0.5,+0.6] = 20% + 35% + 10% = 65%
[-0.1,+1.3] = 35% + 10% + 15% = 60%
This is not what the met office said above.
Also why should the individual probabilities sum to 100%?
Thinking scientist:
Why is the pdf bounded. In my example solution there is a 54% probability of being outside the range.
@Fred Bloggs - the Met presentation was for a layman and not a statistician - that's why they should add to 100, as their idiot-chart clearly does! Except it doesn't.
they state the table does NOT convert NE forecast to UK forecast. why is everyone applying these percentages to that table?
@TS: and they said it s for period nov/dec/jan so you can go ahead and do your stats now but i don't understand why you would.
Fred, my take on this is that the Met Office said there were three
categories (warm, average and cold) and the note obtained by FOI defines
the ranges of these categories. As the probabilities within the 3
categories as stated by the MET office sum to 1 then the implicit
understanding is that the PDF is bounded to the min/max range of -1.5 to
+1.3 degrees anomaly as given by the limits of the categories. If this is
not true then their probabilties must be wrong.
Of course, they must still be wrong anyway, because the categories overlap
and yet the stated probabilities still sum to 1. Fundamentally, the
statement they make appears to have no technical merit on any grounds.
Re: Mark, my understanding is that these are the categories used by
the MET office in the forecast. And why the hell is the MET office, paid
for by UK taxes, so concerned about Northern European averages, and not
making their own UK predictions with their own very expensive supercomputer?
The time period considered does not change the fact that the categories overlap and
yet the probabilities sum to 1.
Anthony
You see? This is why Julia needs her shiny new supercomputer...
Clearly, Anthony is right -- the damn computer is old and dodgy and needs replacement. After all, how long do you expect a steam powered computer to continue?
Still, I once again offer Michael Gallagher as a more economic alternative, who has a proven track record over the last few years. All you need is a few stamps to keep him running. Of course, you will get the forecast for Donegal, but that is near enough.
And Michael can add sums a damn sight better than the Met Office. He needs to to sell stamps, you know.
It's all down to Senna the Soothsayer
Thinking Scientist:
1) How can any model of temperatures be bounded between -1.5 and +1.3? That make no sense and even the met office must know this. I would be aghast if this were the case. I am 100% sure it is not ;-)
2) I provided a simple explanation for why the temperatures sum to 100% above and I was not joking. Things like this happen.
3) Even if the categories overlap, the fact that they sum to 100% does NOT signify a problem with the probabilities as I explained above in my first post.
4) There is no probability density function I can find which can have ALL of its density between -1.5 and +1.3 and also agree with the probabilities given above from the met office.
This whole post may at first sight have seemed interesting and to have exposed a flaw in their probability distribution but when you look more closely it is fine, at least from a pure mathematical standpoint.
A normal (bell shaped) distribution averaged on 0 degrees anomaly with a variance of 3 degrees would give these percentages to one significant figure.
http://i52.tinypic.com/2n0u8au.jpg
So the met office could be saying it will be absolutely normal with a 95% chance of the temperature being less than 6 degrees outside of normal.
Not the most useful of predictions
it's inane scientific blag is what it is.
TS: thanks for headsup. its surprising they send this drivel to the govt.
al:
You are correct to 1 sig fig.
To 2 sig figs I get 40%, 25% and 30% with your Gaussian dbn rather than 40%, 30%, 30%
It's close but I think you need to go bimodal to get a better fit.
al:
The 95% range around the mean is twice the standard deviation (not the variance).
So it is 2 times sqrt(3) = 3.46 degrees and not 6 degrees.
I'll stop now.
@Fred, TS et al
Fascinating. Thankyou.
It's obvious that Mystic Met's superdooper computer picked out three overlapping categories such that the three individual probabilities added up to 1, with the overlapping bits having the same probability as the probability of being outside the lower and upper limits. I reckon Harry wrote the programme to find the category boundaries. ☺
This method of statistics is known as "creative statistics for politicians".
This gives 100% probabilty for each of temperatures -0.1 through +0.4. As you can't get better than that it is therefore a forecast that average temperature T will be in the range -0.1 >= T >= +0.4. All temps outside this range have lesser probabilty. Wiki shows average UK Dec temps as +2.0 to +7.4, so it's a cold weather forecast. Simples.
And it was all a big mistake for a non-scientist Cabinet Office mandarin to interpret this gem as "no clear signal" ...
I'll get my own hat, thank you.
the NE categories in the left column are not bound by the UK 'typical limits' in the center columns. the probabilities should not be applied to this table at all. the 'typical limits' columns are arbitrary 'blag'.
How about normal density with mean 0.077 and std 1.624 ?
no
If you are dealing with spatial distribution of events then probabilities can overlap.
However we are dealing with one event, the mean winter temperature anamoly, on a bounded number line, with overlaping categories. The probabilities make no sense on this basis. Making sense of nonsense by inventing distributions, where probabilties can overlap, you can up with numbers for non-overlaping categories (as I did). But it still doesn't bring clarity.
The Met Office have stated that there was a 1 in 20 (5%) chance of a very cold winter. Why wasn't that categorised and made part of this forecast? Handing out nonsense figures like this to goverment officials means that people are ill-informed - they cannot act and they will not pay heed in the future as a consequence.
For the Met Office's own sake any future forecast for government use should have a series of non-overlaping catetgories that presents a realistic calculation of probabilities over a wide distribution of temperatures. You don't need a supercomputer to do this. That brings clarity and will produce the needed government response required for forecasted events.
It simply beggars belief that an organisation like the Met Office behaves in this way. It beggars belief too that the government don't demand a better service from the Met Office, one that government officials can understand and act on. It also beggars belief that the organisations like the National Grid are dependent on publically available but badly flawed Met Office modelled projections of seasonal weather.
Finally, the government and the Met Office cannot collude to minimise the fall out over this matter. Business have lost heaps of money this winter. People have lost their jobs. A lot of pressure has been put uneccessarily on the emergency services and road, rail and air transport. People have been injured and have died due to the extreme cold.
This is a serious business - it is time the Met Office got serious too.
As pointed out in the post, if there was no temperature anomaly at all, then it would fit the classification of a mild winter, an average winter and a cold winter. If the temperature anomaly was +0.4 degC then it could still be classified as a cold winter. The range of 'cold' is larger than the other categories, so it is not unexpected that it has a slightly higher probability rating. Easy now to adjust equal probability bins of 0.333 each, which look on their face as lacking any skill, to 0.3, 0.3 and 0.4.
There's a lot of comment about what this could have 'meant'. If it's that difficult to figure out, then it's as good as useless.
Still, ambiguous and uncertain utterances kept the Delphic Oracle in business for 800 years.
mark, why not? Seems to fit very well
UC: see my above statement. i maintain the probabilities given in the 'report' for NE weather cannot be applied to numbers given in 'typical upper/lower limits' of the table we're looking at. (although i agree the 'report' is awful).
"Business have lost heaps of money this winter"
Indeed. I seem to recall the chancellor using that as an excuse for (even) worse-than-expected third quarter figures. I was surprised he didn't mention the MO...
mark, I see your point. But for the original question, with pure stats
Event1: -0.1 le x le +1.3, P(-0.1 le x le +1.3) =0.3
Event2: -0.5 le x le +0.6, P(-0.5 le x le +0.6) =0.3
Event3: -1.5 le x le +0.4, P(-1.5 le x le +0.4) =0.4
If x has normal distribution with mean 0.0772 and std 1.6243,
P(-0.1 le x le +1.3) = 0.3177
P(-0.5 le x le +0.6) = 0.2651
P(-1.5 le x le +0.4) = 0.4130
then some rounding and we have the original prediction. Lousy prediction, though ;)
le= Less than or equal