Bishop Hill Voodoo correlations
Apr 15, 2011 Climate: Sceptics Climate: other 
I found this paper via David Colquohoun, who is part of the "official sceptic" world of Simon Singh, Ben Goldacre et al. Despite that, and the despite the fact that it focuses on neuroscience, it somehow seems very relevant to climate science, although I'm too tired to quite put a finger on it at the moment.
A recent set of articles in Perspectives on Psychological Science discussed inflated correlations between brain measures and behavioral criteria when measurement points (voxels) are deliberately selected to maximize criterion correlations (the target article was Vul, Harris, Winkielman, & Pashler, 2009). However, closer inspection reveals that this problem is only a special symptom of a broader methodological problem that characterizes all paradigmatic research, not just neuroscience. Researchers not only select voxels to
inflate effect size, they also select stimuli, task settings, favorable boundary conditions, dependent variables and independent variables, treatment levels, moderators, mediators, and multiple parameter settings in such a way that empirical phenomena become maximally visible and stable. In general, paradigms can be understood as conventional setups for producing idealized, inflated effects. Although the feasibility of representative designs is restricted, a viable remedy lies in a reorientation of paradigmatic research from the visibility of strong effect sizes to genuine validity and scientific scrutiny.
Reader Comments (36)
Surely the paradigm should be one of theoretical, and experimental scientists? That way the theory will be tested by people who don't have a dog in the fight. In climate science we appear to have only experimental scientists trying to prove their own theories. Moreover they're not even experimenting, they're using computers. Computers that told us there would be 50 million climate change refugees by 2010, when there are none, and in fact the areas highlighted as those which people would have to leave are themselves flourishing with growing populations.
Read it and weep that we can't get the message through to the politicians.
http://www.investors.com/NewsAndAnalysis/Article.aspx?id=569069&p=1
".. a viable remedy lies in a reorientation of paradigmatic research from the visibility of strong effect sizes to genuine validity and scientific scrutiny."
i.e. Science.
The problem being that those who sceptics have the most beef with are primarily careerists and not, in any meaningful sense, practicing scientists in respect of the matter of climatology. There is little point arguing with such people for a return to intellectual integrity because theres is a worldview secretly dominated by issues of prestige, tenure and remuneration with little, or nothing, to gain from an honest approach to the work they are nonimally paid to do.
In some cases it is difficult to escape the conclusion that they owe their latter-day career and (ignominious) prominence to nothing more than a shifty and unscrupulous turn of phrase.
You don't have to be tired to find this hard to follow or analyse. Why do some academics write in this complicated gobbledygook which makes everything ten times more difficult to understand than it needs to be?
My theory is that it's to make themselves appear cleverer, and to allow wriggle room in equally obscure phraseology when they are found to be wrong.
@Geronimo "Read it and weep that we can't get the message through to the politicians."
I think Jo Nova's latest post is relevant to this: http://joannenova.com.au/2011/04/professor-points-out-its-a-less-than-nobel-consensus/
The voodoo of verbosity
Yeah, I agree with David C on that one. It's an old problem, especially here in France, where political activists are afflicted with a similar disease. I think it's called jargonitis.
All these endless abstruse and convoluted discussions about scientific method and experimental integrity always take me back to the first lab experiments I did when staring my engineering degree many, many years ago.
The lab supervisor passed out our pristine new lab books, designed with a plain, unlined scribbling page, on the left, for notes, observations and calculations made during the progress of the experiment. The facing page was neatly lined for our results and conclusions to be written up later.
"The most important thing to remember" he said " is that nothing ever gets erased from the left hand page - nothing!".
Feynman's lectures distilled into a single practical sentence.
Didn't anyone ever tell "the team"?
I can discern a blurry figure in the haze of words.
Why do people write like that? Because they can; because clear, precise writing takes time, and because it serves as a kind of masonic handshake by which they can identify themselves to their peers, whose approval they crave. I do it too. I know.
Was that in English?
I refer to this as the Bunchacrooks Effect. A few jail terms for embezzling public funds might improve matters.
There is a rich field to mine in claiming a behaviour maps to region 'x' after staring at 3D regions blobs (voxels) on MRI scans and then using further selective processing through a computer algorithm.
To get an idea how a new pseudo science of "Brain management" could be used for policy and policing the public, I found this interesting story:
The brain police: judging murder with an MRI
http://www.wired.co.uk/magazine/archive/2009/06/features/guilty?page=all
We are going through an age of computer stupidity, where every problem can be allegedly solved by bringing arbitrary computer power to bear. These high profile sceptics seem to be very selective in choosing which sector to criticise when they notice this trend.
Roy
We all do it.
We all do it on here to a certain extent. When words on paper (or on screen) are the only communication you have you invest them with an importance beyond their bare meaning as a vehicle for communicating your self as well as your thoughts.
In my experience those involved in the social sciences (in which I include assorted variations of psychological disciplines, politics, and the communication industry) are the worst, to the extent that sometimes those that are paid to communicate fail miserably to do so!
Think phrenology - a precursor to all the current gobbledegook.
When you get the right BS program running you easily get things like this:
"In respect of specific goals, the product configuration baseline maximises the probability
of project success and minimises the cost and time required for the sophisticated software. On the other hand, the characterisation of specific criteria requires considerable systems analysis and trade off studies to arrive at the philosophy of commonality and standardisation. In this regard, initiation of critical subsystem development affects a significant implementation the structural design, based on system engineering concepts. Thus, initiation of critical subsystem development requires considerable systems analysis and trade off studies to arrive at the sophisticated software. The incorporation of additional mission constraints affects a significant implementation of the anticipated fourth generation equipment."
My spreadsheet can do this all day....perhaps I can get a job with the IPCC. ;-)
I think it’s an interesting paper making a simple point in a complex way. It certainly applies to climate science. With complex natural phenomena, we have to
1. Choose what to study.
2. Decide what parameters are relevant.
3. Decide what data is relevant.
4. Choose some statistics
5. Draft the paper.
6. Improve the paper by looping back to 2, 3 or 4.
7. Publish the paper.
8. Loop back to 3 for another dozen papers.
Grossly oversimplified I know, but there are too many steps involving choice. Complex natural phenomena don’t show you the right choice at any stage, so we get the fatal lure of personal preference making pet phenomena seem more visible than they really are. Cherry-picking is the new paradigm.
The human need to make the experiment and/or the data to fit the theory can overwhelm the best of scientists.
Take a look at Robert Millikan's work on the integral charge of an electron.
One can tell this is an old story you linked, TS (Apr 15, 2011 at 10:26 AM).
The brain police: judging murder with an MRI by Angela Saini. 27 May 09
Today it would read: "The brain police: judging CAGW with an MRI"
Thank you for linking to this, depressing as it is.
Although the paragraph is not shy regarding verbosity, much of it purely for rhetorical effect and open to discussion, it is not particularly obscure.
It looks to me in the line of though where enunciation equal demonstration, which is not the case most of times. I expect that the full paper has more solid proofs of those critics.
Regarding academic gobbledygook, a nasty pandemic disease, you may like this lines from the Patience opera:
If you want to produce some of it yourself with minimal effort, try the postmodernist generator:
http://www.elsewhere.org/pomo/ (you get a different paper automatically generated every time you load the page)
enjoy!
You know what is funny?
David Colquhoun's famous blog - Improbable Science, where he attacks 'complementary and alternative medicine - was (is?) hosted on the University College of London (UCL) servers. Some of the practitioners of said alternative medicine protested vehemently against David's post where he declared their descriptions of the benefits of an herb, red clover, to be 'meaningless goobledegook'.
They wanted the University to not aid him in running his blog by providing server space and successfully had the blog taken down (!!).
And yet, here we are, with David's own description of a simple well-known phenomenon - cherry-picking in science - which reads exactly the same way - utter goobledygook.
I looked at Colquhoun's (and another official sceptic, Steven Novella's, chief of clinical neurology at Yale) recounting of Climategate in their respective blogs. These are people, who will strongly turn against the team and the IPCC, if they are provided, or by some stroke of luck, actually stumble onto how really the vast mountain of IPCC evidence is nothing but a bunch of plausible fallacies, held together by 'catastrophist precautionary principle' rubberbands.
Colquhoun, for example, says so many right things, but this is offset by completely bizarre and repeated references to Sarah Palin. Many, in the official sceptic crowd, stay inside inside their cosseted boundaries as far as AGW, merely because it seems, they fear that if they examine its case they might appear to themselves as similar to 'Sarah Palin'.
The article above falls in the same category: herbal medicine dudes, Sarah Palin: evil idiots. Establishment Science: striving for eternal truth, their methods to be mollycoddled and papered over by wordy descriptions.
Colquhoun article on Climategate and CRU
Whaaaaat? What language is that? Double Dutch!
And, if you get past the language, it does sound like one phrenologist arguing with another!
George Orwell, Politics and the English Language:
"I am going to translate a passage of good English into modern English of the worst sort. Here is a well-known verse from Ecclesiastes:
I returned and saw under the sun, that the race is not to the swift, nor the battle to the strong, neither yet bread to the wise, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all.
Here it is in modern English:
Objective considerations of contemporary phenomena compel the conclusion that success or failure in competitive activities exhibits no tendency to be commensurate with innate capacity, but that a considerable element of the unpredictable must invariably be taken into account.
This is a parody, but not a very gross one...It will be seen that I have not made a full translation. The beginning and ending of the sentence follow the original meaning fairly closely, but in the middle the concrete illustrations -- race, battle, bread -- dissolve into the vague phrases "success or failure in competitive activities." This had to be so, because no modern writer... would ever tabulate his thoughts in that precise and detailed way. The whole tendency of modern prose is away from concreteness. Now analyze these two sentences a little more closely. The first contains forty-nine words but only sixty syllables, and all its words are those of everyday life. The second contains thirty-eight words of ninety syllables: eighteen of those words are from Latin roots, and one from Greek. The first sentence contains six vivid images, and only one phrase ("time and chance") that could be called vague. The second contains not a single fresh, arresting phrase, and in spite of its ninety syllables it gives only a shortened version of the meaning contained in the first. Yet without a doubt it is the second kind of sentence that is gaining ground in modern English. I do not want to exaggerate. This kind of writing is not yet universal, and outcrops of simplicity will occur here and there in the worst-written page. Still, if you or I were told to write a few lines on the uncertainty of human fortunes, we should probably come much nearer to my imaginary sentence than to the one from Ecclesiastes."
http://www.mtholyoke.edu/acad/intrel/orwell46.htm
AIUI, what the windbag who wrote the following passage ("However, closer inspection reveals....scientific scrutiny") in the OP meant was:
When you look closely, this problem seems like just one symptom of a wider one with all research into established theories, not just neuroscience. Researchers design experiments so that some or all the usual variables – conditions, parameters, and so on - tend to give you whatever result you were expecting. In a way, experiments are managed so as to shore up whatever you already knew. There is not much one can do about this, but a start would be if researchers designed experiments rigorously enough that the theory under test could fail.
I think that says the same, but my version is 91 words and 143 syllables rather than 113 and 270 syllables. The only serious elision I made - stimuli, task settings, favorable boundary conditions, dependent variables and independent variables, treatment levels, moderators, mediators, and multiple parameter settings - was only stuff that his audience can presumably populate from their own knowledge.
It's kind of shocking that 65 years after Orwell wrote his piece it's still happening. I wouldn't expect a highly technical article on neuroscience to be wholly jargon-free - but this piece is caked with it and isn't technical!
I actually think the abstract quoted at the top doesn't do justice the paper, they seem to have boiled it down to an almost abstract minimum there.
I think the paper is worth reading in full. It is fascinating to see a meta-discussion like this within cognitive science, the motives of the science practioners themselves are discussed in the same language as the science. Although there are suggestions in the paper that this "voodoo" tendency isn't restricted to neuroscience.
There are a lot very easily understood and very quotable bits. My cognitive state makes me cherrypick this one :)
"...people and organisms tend to continue sampling as long as it is pleasant, but they stop when it becomes unpleasant. Accordingly, researchers continue to use strong setups but truncate the use of setups that yield weak findings."
And one of the recommendations in the conclusion is
"Unrestricted sampling in broad-minded meta-analyses may indeed reduce sampling biases and create new insights."
this seems to speak about the sort of behaviour we see in the cherry picking within paleo climate constructions; and points to the motives behind the fierce criticisms of the auditing behaviour of McIntyre, who look at "too" much of the remaining data ;)
What's just come to mind is the title of an old Radio 2 programme: Singh Something Simple....:o)
The original paper by Vul et al, about fMRI can be downloaded here:
http://www.edvul.com/publications.php
It is an easy read, and pretty damning. It contributed to my increasing sense that climatology is typical of a lot of modern science.
A particular striking observation in that paper, is that certain emotional states are assessed by questionnaire, and this technique is only repeatable from test to test with a certain percentage accuracy. In some cases, the reported fMRI correlations are more accurate than the test that is supposed to identify a given trait!
"If you can't dazzle them with brilliance, baffle them with BS."
Don Pablo
[OT, with apologies to all:]
You were asking recently about ice cores and paleoatmospheric CO2. I think J provided links, but I came across this article the other day which may interest you. One caveat though - read the comments from Ron Broberg and his response at the Whiteboard (link in comments):
http://debunkhouse.wordpress.com/2010/12/25/co2-ice-cores-vs-plant-stomata-wuwt/#more-617
The quoted passage is from the paper and not Colquhoun's. Sheesh. :)
@A K Haart
Excellent summary. The only minor addition would be:
5a copy-and-paste from old papers repeatedly (to avoid over exerting ones fragile creativity, and elevate ones citation index score artificially)
Seems relevant!
Let the scanning commence...
http://www.guardian.co.uk/politics/2010/dec/28/political-allegiances-brain-structure-study
An experiment has found differences between the brains of progressives and conservatives.
Head scans of students at University College London, conducted by neuroscientist Geraint Rees, showed a "strong correlation" between thickness in two regions, the amygdala and the anterior cingulate, and political viewpoint.
Rees said he was "very surprised" by the finding because the experiment was a lighthearted item commissioned by Colin Firth for his guest-editing slot on BBC Radio 4's Today programme.
The actor has said he no longer supports the Lib Dems, and joked about Nick Clegg: "I think we should have him scanned."
BBD
Looking at the commits I find:
"Tis all BS"
How the freck do they measure what happened 1000 years ago with such accuracy? I think this is right in line the present topic of obfuscation in the name of post normal "science.
Instead of playing at scientists, their time would be better spent at being scientists. I have no faith that 1000 year old ice has anything like the chemical composition it had when it was formed. And as for the magical carbon isotope separating "features" of plant stomata, I also am more than a little dubious.
Good question Don Pablo. Have you ever read Michael Crichton's speech on "ALIENS CAUSE GLOBAL WARMING"? He points out the abuse of science by scientists by quoting with such certainty things that they cannot be certain about. It's pretty famous, so you probably have, but if you haven't it is highly recommended.
http://scienceandpublicpolicy.org/images/stories/papers/commentaries/crichton_3.pdf
Don Pablo
It was FYI and off topic. I endorse nothing;-)
Bishop's Finger....tiredness.... I recognise the symptoms, all too well.
Let me have a go then : setting up 'tests' and 'science' to buttress a predetermined position ?
I'm beginning to think that part of the problem is the R package which most of the climate models seem to be built in -- because it is so powerful.
For statistical analysis, I can build a better model in 6 lines of R than I could in 600 lines of a general-purpose language like Ruby or C.
That means 1) I can create hundreds of different models very easily and 2) I never really need to look into the guts of what I am doing to see whether it makes sense or whether my initial assumptions are reasonable, let alone correct.
For example, a moderately complex forecasting process called the Estimated Logistic Regression Equation can be implemented in just 4 lines of R, meaning that I can generate plausible results without having much idea of what I am doing.
Sometimes the slower ways are better, perhaps.
Rick Bradford
You make EXCELLENT points: Garbage in, garbage out, all brought to you through the wonder of modern technology and computers.
BBD
No, I think it was on target -- just from a different angle. As you might guess I find most of the "proof" on both sides to be questionable, but it should be looked at and discussed. I appreciate you raising the point.
TakuanSoho
Thank you -- I had missed this.