Monday
Jul182011
by 
Bishop Hill
Bishop Hill The bulldog that didn't bark
Jul 18, 2011 Climate: Statistics Climate: Surface Tamino has been looking at the question of statistical significance in the temperature records. The good news is that he appears to agree with many on the sceptic side of the debate that AR1 is not a suitable model - it's always nice to find some cross-party consensus, particularly when this suggests that the IPCC is wrong. Tamino's preference is for an ARMA(1,1) model.
However, Doug Keenan emails to say that he has been trying to leave a comment suggesting another model:
The statistical model used above is a straight line with ARMA(1,1) noise. I do not know of a good justification for that model, and it is easy to find alternative models that have a far better statistical fit to the global-temperature data.One alternative model is fractional Gaussian noise (also known as "Hurst-Kolmogorov"). Good justification for fGn has been presented by Demetris Koutsoyiannis. In particular, fGn arises as a consequence of the second law of thermodynamics [Koutsoyiannis, Physica A, 2011]. The AIC value of fGn is also far lower than that of the model used above—you might check this for yourself. For more details, see the post at Bishop Hill:The fGn model has no trend.
Unfortunately, this doesn't seem to have made it through moderation.
Ho hum.
(Peter Gleick is also riffing on statistical significance, pushing Phil Jones' recent claim that the trend since 1995 is significant - we never did find out what basis this had in fact).
Reader Comments (11)
The link in Mr. Keenan's comment is giving me a "page not found" here is the correct link: http://bishophill.squarespace.com/blog/2011/6/6/koutsoyiannis-2011.html
I looked at ARMA(1,1) applied to various series in 2005 - see www.climateaudit.org/tag/arma_1_1.
People have to be careful as to what they think they understand when something is of no "statistical significance". Observed trends are about halfway between no trend and the model ensemble. If one uses fgn, it fans out the distribution so that no-trend is no longer in the 2.5% percentile. But it's a two-edged sword. It also fans out the upper limit as well so that 2.5% upper percentile is much higher under fGN as well.
@ FergalR
I sent the link as typed, but the system seems to have corrupted the underlying hyperlink.
@ Steve McIntyre
As stated in my message, the fGn model has no trend. Hence there are no confidence intervals for a trend. You are looking at some other model, not fGn.
I've always struggled with the idea of statistical significance and confidence especially as it is applied to temperature time series.
The original idea was I believe to ensure that the effect being sought was actually there rather than the result of chance. The 95% confidence level was introduced by Fisher as a rule of thumb rather than the gold standard that it has become. This is probably the reason why so many drugs have to be withdrawn despite achieving P=>95%. It's still a 5% chance of being by chance which is riskiness that few can tolerate in normal life - how about a 1/20 chance that you will crash your car?
I am also puzzled by the ability of statistical torture to get significance from measurements that are known to carry errors that are larger than the effect being measured etc etc.
Can somebody help a simpleton like me?
Regards
Paul
P.S on Material World on R4 tonight there was an interview with a BAS scientist who has been looking for volcanoes on the sea bed in water off of Antartica. I didn't catch the location but in what sounded like a relatively small area they found 12 previously unknown active volcanoes two of which had crests 50 and 70 odd metres below the surface. Sounds like vindication for Plimer.
Paul, I too find it somewhat amazing the degree of confidence possible to be extracted from the underlying data. To me its not only a problem of inaccuracy at source, its also that such measurements are often subjected to operational and political 'adjustment' to suit the needs of the time. Combine that with the fact that I have yet to find a model that carries through the maths the monthly mean and SD per data point through the processing - without that you have no way of knowing if a given months average temperature is based on a fairly constant figure or something highly erratic.
Given that I run the NASA GISS model on my quad core unix box in under 10 minutes - a possible max x6 increase in processing load (figure you need the average, sd and pop carried through) should mean only a one hour wait to carry through direct significance figures per resultant data point - i.e. you could plot significance onto the globe, be interesting to see what the poles show...
@ Paul Maynard
95% confidence is just two standard deviations assuming a normal distribution of data points, of course.
That is, if you have a range of normally distributed values - the height of 10,000 adult females, say - and you graph them you will get a "normal" i.e. bell-shaped distribution curve. The area underneath any curve integrates to 1. If you calculate the standard deviation you can mark them as verticals on the chart, one on either side of the mean. The area enclosed between the lines, the curve, and the x axis will be 67% of the total area under the curve.
The area underneath the curve between the + and - 2 standard deviation bars is ~95% of it. 3 SDs = 99.7%, IIRC.
In practical terms all that 1 SD (or "sigma") means on the chart is that 67% of the data points fall inside those bounds. 95% fall inside two. And so on.
Apologies if I am insulting your (or anyone else's) intelligence here.
All 95% confidence means, as far as I can judge, is that the sequence of temperature readings since 1998 is greater than 2 standard deviations away from the mean, i.e. outside the ~95% of the graph bounded by the standard deviation bars.
Call me old fashioned but that's not a specially testing threshold. When the hedge fund Amaranth fell over 5 years or so ago, the event that killed it was a nine-sigma market movement that day. Now 9 SDs is a seriously outlying result, one so improbable you'd barely plan for it. If you're talking daily events then a 6 sigma event in fag packet terms happens one day every 4 million years. In fact once you get up past 6 sigmas things like Excel and even Matlab can't handle them.
A 2-sigma event in contrast happens about 5% of the time. So in ballpark terms if, I don't know, daily FTSE movements follow a normal distribution, you'd expect 12 to 13 2-sigma price outcomes every year. In other words, while 95% confidence sounds impressive, I surmise that it is mistaken for 95% certainty by arts graduate ministers who've only ever heard the term used colloquially, and don't know what it means.
I further surmise that this is not an accident, and that terms like it are bandied around by climate psyentists to add deliberate opacity and a patina of wholly spurious gravitas to what is, in fact, rather simple data-collecting and number-crunching. This is done at crappy provincial universities by rather stupid second-rate people with no special expertise in statistical techniques.
I also tried to leave a comment on Tamino's website. It was 'moderated' and did not make it.
Excuse me if I'm wrong, but it appears that the tests that are being used and recommended require the assumption of normality- normal distribution of independent data points.
Climate is non linear, chaotic - the tests are not appropriate.
In fact, you cannot forecast climate unless you know ALL the variables, how they relate to each other EXACTLY and their initila values down to the thrillienth decimal place.
So what's the point of error bars?
@ AusieDan
Well, yes, quite. You have to know how your dataset is distributed a priori before you can say very much more about it.
@ Paul Maynard
Some discussion of what “significance” means for a trend is given in an article that I published in the Wall Street Journal last April.
About errors in the measurements, assuming that the errors are essentially random, then they will tend to cancel each other out. For example, suppose that all the errors have a Gaussian distribution with mean 0 and variance σ²; then the variance for the average of n measurements is σ²/n. Thus, by taking a large number of measurements, the variance of the average can be made extremely small, i.e. the average can be made extremely precise.
@ AusieDan, Justice4Rinka
Tamino does indeed just assume that the underlying distributions are normal/Gaussian (as does the IPCC). That is not, though, what I was doing here. Rather, I was taking two statistical models of the data and comparing their fits.
It turns out that one of the models fits the data far better than the other (comparing via AIC). The model that fits far better, fGn, also has the advantage of being derived from basic physical principles (especially the second law of thermodynamics).
_______________________
One of the issues here is that Tamino is apparently a time series analyst. Hence he must understand that what he posted on his blog is wrong. Yet he rejects comments pointing that out.
[No]