Friday
Feb032012
by 
Josh
Josh Cherrypicking
Feb 3, 2012 Climate: Sceptics Climate: Statistics Josh There has been a lot of blog battling and twitter twootling started by this WSJ article
with climate crossness here and here and here, all brilliantly buffed and rebuffed by Matt Briggs
A lot of the argument was about 'how to do climate graphs' with input taken from Skeptical Science.
H/t MrSean2k at WUWT who finds a rather good example.
So here is a quick lesson in how to do Climate Graphs - please concentrate.
;)
From Cartoonsbyjosh.com (click image for larger version)
Reader Comments (144)
But I thought if you were a climate scientologist, you just drew the line wherever you wanted it to go, and then accused anyone who disagreed of heresy.
Nice one Josh!
May I submit my poor attempt to improve on the IPCC figure?
Of course I shun short trends ;-) Here 110y and 60y are compared. Same old same old, though...
Curious that Matt Ridley's piece in the WSJ 'Are We Holding a New Ice Age at Bay?' on Jan 14, (with full graphs on the Rational Optimist blog under the title 'The slow cooling of our interglacia'l- see sidebar link) did not invoke equivalent alarmist wrath. Perhaps because they would rather not focus attention on inconvenient longer time scale perspectives. It does rather make the modern temperature rise rather trivial, after all. That would never do.
BBD, Ooh! thanks for the link now we can all play picking trends, I found global warming too from 1900 to 1950 but only for 50 years :(, there was no plateau to tack on the end ;)
Try graphing the slopes derived from calculated linear trends of moving averages, 13 months, 10 years, 30 years and 50 years for the various instrumental records. See what you come up with.
Mac
I might be getting my wires crossed here, but didn't Matt Briggs counsel very sternly against doing just that?
Which slope was caused by CO2 and which wasn't?
The shortest shallowest one of course.
At 6:00 AM today the ambient temperature was -- 16.5 degrees Celsius
At 10:00 AM today it was -- 19 degrees Celsius
So this time next week the ambient temperature should be -- 124.5 degrees!
Can I have my grant now please?
BBD - I think that any real statistician would say that the smoothing technique should reflect physical principles, or at least some assumption a priori as to how the trend should appear. It is not enough to take data and then impose a trend on it and claim theoretical mastery.
It seems totally absurd to me that climate science chose the arbitrary average surface global temperature metric as the definitive absolute reference for the elucidation of global climates. It is an arbitrary synthetic measure with no tangible relationship to the real world.
Only 29% of the Earth’s surface is land. The bulk of that land, and roughly 90% of its human population, live in the northern hemisphere. The land area lies at variable altitude, and therefore temperature measurements are complicated by unevenly spaced measurement records of widely variable quality, complicated not only by the adiabatic altitudinal lapse rate but also poorly quantified urbanisation effects.
Almost nobody lives in the Arctic, most live either in the temperate zone between the Arctic Circle and the Tropic of Cancer, or in the tropics. Climates in the northern temperate latitudes are dominantly controlled by the clockwise-induced airflow cells and consequent ocean current circulation. The majority of meteorological stations are located in this 29% of the Earth’s surface. The recent temperature history of the northern hemisphere differs greatly from the southern hemisphere.
Approximately 71% of the Earth’s surface is ocean, and long term measurement records are almost non-existent.
Attempting to populate a long range temperature grid of the world from this database is dubious enough, but to think that is has any application to the furtherance of proper understanding of climate science defies logic.
Real data measurement and proxy time series of considerable duration are however available from numerous specific locations, and it is only by reference to those regional entities , and to the interpretation of the mechanisms responsible for their specific climate histories, which will advance understanding in any practically applicable way to that region.
The average global temperature measurement is in my opinion irrelevant to advancing the understanding of climates.
It is merely a political expedient, and the fact that so much emphasis is placed on it tells me that the climate community are more interested in the political agenda than in the furtherence of pure climate science.
Diogenes
Not sure I follow you. The smoothing technique is just averaging - no trend is imposed. Nothing is imposed.
This is HADCRUT3 (monthly averages)
This is HADCRUT3 (annual averages)
In the second example, HADCRUT3 monthly data are averaged to give an annual average. The change in annual averages is easier to interpret than the 'noisy' monthly data.
If monthly data is averaged over a five year period, the smoothing is greater (nothing has been added) and the result is clearer still.
BBD...a priori, what do you think the data should say.
Have the data said that?
I thought you were scientific.
BBD, this pivot tells us that 1998 was a watershed year in terms of climate on land.
It says the short warming period starting around 1980 came to an end.
"The smoothing technique is just averaging - no trend is imposed. Nothing is imposed."
You should actually spend a little time trying to understand the theory behind what you are saying. It's pretty clear you do not have any clue what regarding what you are saying.
Mark
hint: discrete time signal processing and linear systems theory are two good search phrases.
MarkT
If averaging as a smoothing technique is problematic, why is it in universal use?
The headline refers to cherry picking. I fail to understand why sheptics are concerned when warmists accuse the skeptics of cherry picking. It appears to me that the concept and use of cherry picking is misunderstood.
One can never legitimately chery pick data and claim that that data (ie., the cherry picked data) proves the correctness of the theory. At most, it is consistent with the theory but beyond that, it tells us nothing about the validity of the theory.
However, one can legitimately cherry pick data or scenarios to establish that there is a potential problem,with the theory. Indeed, most theories are disporved in this manner. One could cite millions of scenarios that would seem to suggest the correctness of Newtoniam mechanics. However, it is in the cherry picked extremities that one can see that it is not universally correct.
Accordingly it is a legitimate scientific tool to cherry pick part pf a data set and say this appears to be at odds with the theory. If the theory is sound it will have a valid answer (which is consitent with the workings of the theory itself) which will be able to explain what is observed in the cherry picked data series.
It is not cherry picking to say that there has been no warming since 1995 or 2000 (or what have you). This is simply a statement of fact upon which the theory needs a valid explanation. That explanation may be that short terms trends in natuiral variation over come the warming effect of CO2, or it may be that during the extrated preriod the CO2 warming trend is countered by an equal and opposite aerosol cooling trend and/or a combination of both (or even some other legitinate explanation). However, it is a wholly deficient response for protagonists of the theory to simply say you are cherry picking and therefore the poit you raise is not relevant thereby not offering an explanation for the selected data. In short, if a theory is valid, it should be capable of dealing with any data set that is thown at it.
Finally, I would just point out the obvious that in climate science one is always cherry picking data unless one presents a full picture dating back to the date when the planet first aquired an atmosphere. Looking at the past 100 years is in practice little better than looking at just the past 10 or 15 years. Both are incomplete.
digenese
I'm still confused. I don't think the data should 'say' anything. Smoothing can help reveal an underlying signal. I don't understand why this is contentious.
CO2 is bad. Very hard on instruments. It caused the early temperature readings to be too high and the more recent readings to be too low. That is why graphs need to be fixed by experts. Because of CO2.
BBD has just confirmed my last post. The "smoothing" is required to "reveal" the "underlying signal" that only the IPCC approved experts can see.
Similarly, the emporer's clothes need to be ironed.
You must not be able to read, either. Try it again and see if you can respond to what I said, not what you think I said. And, for the record, the universe of signal processing extends far beyond climate science. Again you make claims regarding things you clearly have no knowledge of, yet seem comfortable claiming people in here are all wrong about nearly every topic.
Mark
Mark T
Please, explain why averaging to reduce the signal/noise ratio is a problem.
Again you ask a question that does not address anything I said. Try again.
Mark
Mark T
This is daft. Averaging to reduce the signal/noise ratio is not a problem. Hence its universal application. Why should its application to a temperature time series be inappropriate?
If there is one thing that we can learn from this debacle and therefore bequest a true inheritance to those that follow, it is that we put in place irrevocable systems that protect and enshrine the RAW data.
That we hand it on unmolested, not smoothed, not adjusted, not refined, but archived just as it is.
If we manage this, which let’s face it is a simple task, then those that follow will have a known base upon which to make their “informed” decisions.
This would reduce their need to follow their forebears in constantly revisiting and revising history.
At present we are possessed with a number of people that are totally sure that our understanding of history is wrong and totally convinced about what our future holds, yet are totally bereft of an explanation about what is happening today.
Read what I actually said. You still have not figured that out. Maybe then you will learn something. I'll give you a hint: I never said anything about a problem, you merely assumed that because a) you really do not understand anything about filtering and b) you know it, but cannot admit it for fear of revealing your lack of understanding.
Mark
The last 15 years is part of a wider issue.
The 'basic' physics upon which the GHE is based dictates that whenever there is an increase in CO2 this inextricably leads to an increase in temperature; the backradiation from increases in CO2 does not allow for a stasis in temperature still less a reduction in temperature, it is a one way street leading always and only to an upward path in temperatures.
Given this 'basic' physics, the real and wider issue is how do the protaganists behind the GHE claim that an increase in CO2 leads to a temperature increase when this is not borne out by the attached graph?:
http://www.woodfortrees.org/plot/hadcrut3vgl/from:1860/to:1880/from/plot/hadcrut3vgl/from:1881/to:1898/plot/hadcrut3vgl/from:1899/to:1942/plot/hadcrut3vgl/from:1943/to:1958/plot/hadcrut3vgl/from:1959/to:1996/plot/hadcrut3vgl/from:1997/to:2011
This graph, not just the last 10 or 15 years of it, does not support the contention that there is a correlation between CO2 and temperature.
When shown this graph, those standing behind the theory need to explain the 1860-1880 warming, the 1880 - 1900 cooling, the 1900-1940 warming, the 1940-1960 cooling, the 1960-2000 warming, the 2000-2011 stasis (approximate dates). This in no way corresponds with levels of CO2 in the atmosphere. As regards the 1960-2000 warming the expanation needs to explain why the rate of warming during this period is not markedly greater than the rate of warming during the 1900-1940 warming.
The last 10 or 15 years is all part of the same wider issue, namely that based upon the modern instrument record there appears no correlation between CO2 levels and temperature.
The link of the referenced plot should read:http://www.woodfortrees.org/plot/hadcrut3vgl/from:1860/to:1880/from/plot/hadcrut3vgl/from:1881/to:1898/plot/hadcrut3vgl/from:1899/to:1942/plot/hadcrut3vgl/from:1943/to:1958/plot/hadcrut3vgl/from:1959/to:1996/plot/hadcrut3vgl/from:1997/to:2011
Further to my post at 12:02 am, I am having difficulty attaching the link to which I want to refer. Accordingly, I am breaking this down into five parts.
Please noite that the link of the referenced plot should read in one continuos line:
http://www.woodfortrees.org/plot/hadcrut3vgl/from:
1860/to:1880/from/plot/hadcrut3vgl/from:
1881/to:1898/plot/hadcrut3vgl/from:1899/to:1942/plot/hadcrut3vgl/from:
1943/to:1958/plot/hadcrut3vgl/from:
1959/to:1996/plot/hadcrut3vgl/from:1997/to:2011
It is therefore necessary to cancel the paragraph break at the end of each line.
Sorry for the inconvenience.
Mark T
Please try to avoid snark in your responses to BBD.
Feb 3, 2012 at 11:58 PM | Green Sand
//////////////////////////////////
I second that.
richard verney
Thanks Richard, get your drift. Things change, I think we all know and accept the physical props of CO2, what we don't know and don't look like knowing for a long time is what this planet decides to do with the products of said gas, or for that matter any other form of potentially recycled energy.
PS, if it helps the link worked for me in your first comment, I highlighted it and right clicked “open in new tab”
"Please try to avoid snark in your responses to BBD."
Yes. He's a delicate flower.
;)
Andrew
"Please try to avoid snark in your responses to BBD."
Why? BBD treats everyone with contempt. He is not interested in learning, even regarding a topic he knows his knowledge is incomplete. Were he interested, his first response to me would have been:
"You are correct, I do not have any real understanding of what smoothing does because I have never studied it. Can you please tell me how my claims were incorrect?"
Instead, he responds in a manner that indicates he either did not read my original statement or was unwilling to admit his deficiency. Three times and he has still refused to say anything other that "what is the problem with smoothing." Three times - it was not me being daft, clearly BBD did not want to address what I actually said otherwise he would have.
Every one of these statements is fundamentally incorrect:
1. "The smoothing technique is just averaging - no trend is imposed. Nothing is imposed." (to which came my first reply).
2. "If monthly data is averaged over a five year period, the smoothing is greater (nothing has been added) and the result is clearer still."
3. re: smoothing, "... why is it in universal use?"
4. "Smoothing can help reveal an underlying signal. I don't understand why this is contentious."
Rather than making claims regarding topics BBD has simply lifted from others, it would be nice if he would actually make an attempt to verify his own understanding once in a while. I gave him an opportunity and he chose to avoid it. Instead, he has proved that he does not want to learn (he thinks he already knows enough) and is unwilling (or unable) to admit the areas in which his knowledge is lacking.
Mark
Bad Andrew
"Yes. He's a delicate flower."
And that is probably one of the most accurate comments ever posted on this blog.
hint: discrete time signal processing and linear systems theory are two good search phrases.
I did this, in order to see where BBD is wrong. After an hour of apparently wasted time, I still don't see where this shows BBD is wrong.
All your snarky little comments have done is make me side with BBD on this issue. Which I doubt was your intention.
Now I'm not a troll. I'm not a warmer either. So tell me what is so awful about using an properly centred average to smooth a sufficiently long time series.
Richard Verney: thank you for everything you've written here but especially putting to rest the false fear of cherry picking. Einstein was indeed a terrible cherry picker. His Quaker admirer Eddington didn't just point his instruments anywhere but travelled to a total eclipse on Principe to seek out the first raw data that would make clear that Newtonian physics was wrong, even if it took a while, on the back of much more data, for general relativity to be widely recognised as its replacement. (And that conservatism is also quite proper, as Pat Frank has pointed out, except I suppose in an area of science which has become thoroughly corrupted. And that's the rub.)
Mark T
Just ignore him. You'll get used to it after a while.
I used to like the guy to be honest. But he has been on a downhill for some months now.
He lost the last shred of credibility he had with me about a week or so ago, when he replied to Nullius in Verba, who had patiently given him a comprehensive reply on some issue, with a completely unwarranted and snarky single-sentence 'rebuttal': "sometimes I think I am the stupid one here, but.."
Since than I have completely ignored every single comment he has made.
You can learn to ignore him as well.
The problem with averaging is simple - the wider the average period the flatter the curve, so at what point has the person choosing the period width gone too far? A plot of the last hundred years using 100 years as the period width will plot a single point. So a great deal of meaning has been lost, and the trend will be 0. For moving averages, the greater the MA parameter, the more sinusoidal the curve (see Slutsky-Yule). A trick I have seen in business presentations is to control the trend line by adjusting the MA parameter. If you want a flat/steep trend then you crank up/down the smoothing, and then trend the smoothed points. This is completely wrong, of course - the linear trend should always be of the unsmoothed data points - but it means that if persuasion rather than truth is the objective, trends can be pretty much anything you want.
"I did this, in order to see where BBD is wrong. After an hour of apparently wasted time, I still don't see where this shows BBD is wrong."
Wow, you digested all of linear system theory and discrete time signal processing theory in an hour. I never said that you'd find ALL of the answers just by doing a search, I said those are good search phrases. That's a pretty good hint that the theory behind why BBD's statement are incorrect is a little bit more involved than a simple google search.
This is the problem with people in this debate, they seem to think they can simply enter some search terms into their browser and oila!, magically they understand the very stuff that takes years for the true "experts" to learn in the first place. I gave a starting point, you failed to see it as that.
"All your snarky little comments have done is make me side with BBD on this issue."
Good for you. Those that choose to be ignorant deserve the scorn they receive as a result. Join BBD's club I suppose.
"Which I doubt was your intention."
I couldn't care less what you think. I'm right. Sorry if that makes you all butt-hurt, but it's the truth. Somewhere along the line the masses arguing over these "nits" (to them, not me) need to accept the fact that they don't really understand what it is they are doing every time they apply some technique to data. The simple fact is that things like filtering seem simple, but they are decidedly not. That's why there are PhDs awarded in the areas: they are complex. Climate scientists, in general, are NOT properly trained yet willy-nilly apply the art as if they are with (often) dire consequences.
"So tell me what is so awful about using an properly centred average to smooth a sufficiently long time series."
Man. I stated this FOUR FREAKING TIMES! Not once did I use the word "problem." Any potential "problems" have nothing to do with what I took exception to. Are you people blind or just illiterate? Unbelievable.
Mark
"Just ignore him. You'll get used to it after a while."
I know who BBD is quite well as I've been reading his idiocy for at least a few years now. Since before his conversion as I recall, and he did a good job of demonstrating his unwillingness to learn even then. Sometimes people that see their domains getting trashed need to make a point.
"I used to like the guy to be honest. But he has been on a downhill for some months now."
Longer than that.
;)
Mark
Give me a few minutes and I will explain what BBD was wholly incapable of asking for. To people like me, this is basic stuff, the fundamentals, but I do realize this particular topic is not something you'll simply pick up. Most of the application I've seen from the climate community is just awful.
Just so you know, my first introduction to the concepts came from my first linear systems theory class in 1988 or so. We were required to take two of them (not all EE programs do*), and I followed those two classes up with about 80 others. System theory is the backgrounder for understanding just about everything involved with signal processing (control theory, too, which helps with signal processing as well).
Mark
Oops, the * was for: Signals and Systems, Ziemer, Tranter and Fannin. The one listed at Amazon seems different than the one I used as an undergrad. All three of the authors were advisors of mine at one time or another though none of them taught the classes I took.
Mark
1. "The smoothing technique is just averaging - no trend is imposed. Nothing is imposed." (to which came my first reply).
This is patently false. First, as soon as you apply a filter (or any linear transformation) to data you have immediately imposed a model on the data. In fact, you have imposed not just a "signal" model, but also a "noise"* model. This point is not contentious. It is flatly true, and sufficient to prove that my first objection was justifiied.
Furthermore, had BBD actually attempted to understand, or had he simply asked someone that does know before replying with his strawman, he would have known that the correctness of his statement had nothing to do with there being a problem. Instead, he chose the path of ignorance and replied defensively, as if the fact that I know more about this than he does represents some injustice in the world. Give me a break.
There's more, but on to the next point first and I'll wrap it all up with some of the "problems" (yes, there are plenty, some of which can be gleaned just from the above statements).
Mark
* In general, for purposes of that statement, "noise" simply refers to anything that is not part of the desired signal. You can go round after round debating what noise really is, which is largely immaterial to my point above.
Mooloo:
Sounds like you cherry-picked the wrong bits of 'linear system theory and discrete time signal processing theory' in that hour, which I find faintly ironic. That's what Mark is saying anyway - and, without being willing to do the PhDs required, I believe him.
Mark
The stuff has been explained to BBD in various forms - by PaulM, by me, by you etc.
PaulM said:
I said:
That thread ran for >500 comments. I'm sure you've seen it. The situation is simply hopeless.
2. "If monthly data is averaged over a five year period, the smoothing is greater (nothing has been added) and the result is clearer still."
There are several reasons this statement is incorrect.
First, the parenthesis hold a falsehood, though this is minor. You have added something: you've increased the number of points in your estimate of the "true signal." Second, the "result" is not clearer from this, it simply has fewer high frequency terms. You've applied a signal model that assumes low-frequency content, then modified it to be even lower frequency (increasing the length of the average). If the "true signal" actually lay somewhere between the two cutoff frequencies, then you've actually muddied your result by removing legitimate signal.
Mark
3. re: smoothing, "... why is it in universal use?"
It is only in "universal use" by climate scientists. Your reference point is somewhat suspect since they really are not trained to deal with these things.
I suppose I can't say only climate scientists. Indeed, moving averages (as well as the integrate and dump) are used rather regularly by signal processing folks. In fact, integrate and dump is used all over the place in a modem, as an example. The difference, however, is that we typically have a priori knowledge of the signal content and know how to properly apply such filters and understand what the "results" mean.
Mark
4. "Smoothing can help reveal an underlying signal. I don't understand why this is contentious."
It is contentious because it is wrong, or at least, misguided. This is certainly the weakest of my objections since, given knowledge of the signal, it can actually do this. However, as I noted previously, if the signal lies outside of the bandwidth of the smoother, you will not reveal any underlying signal. You will attenuate it.
For some reason there is simply this death grip like hold on the belief that all signals are decidedly low-pass, indeed, that the entire climate system can be modeled by a freaking straight line (essentially DC). Absolutely ridiculous.
Mark
"That's what Mark is saying anyway - and, without being willing to do the PhDs required, I believe him."
Thanks, Richard. I was stupid enough to do them, while working full-time, with a working wife and 4 1/2 year-old son (at the time, almost 9 now). Needless to say, my wife and I will be parting ways soon - that was just too much for our marriage.
"I'm sure you've seen it. The situation is simply hopeless."
Oh yeah. I may have even posted in it.
Mooloo: I agree that my first statement regarding the search was perhaps a bit obtuse, indeed, I was being intentionally facetious, though I was serious that the theory starts there. It was broad to prove a point that the reasons why these things are thought to be true (technically still theoretical) is much more complex than people realize.
In general, an undergrad engineer (most take some system theory) will likely be able to figure out these things. Anyone that goes on to an advanced degree focusing in one of several areas will necessarily be given deeper details into the theory.
I'll finish with the problems tomorrow. I'm tarred and ready for the couch. Wife gets the bed. :)
Mark
Mark T,
It is not just that "that the entire climate system can be modeled by a freaking straight line "... the line must be positive.
If the climate system is modeled and shows a negative line, then that truly is an incorrect way to process the signal.
See, the correctness of the method is dependent on the slope itself, not the actual method used.
At least, that is what I have understood from all the various blog rebuttals so far.