Voodoo correlations
Apr 15, 2011
Bishop Hill in Climate: Sceptics, Climate: other

I found this paper via David Colquohoun, who is part of the "official sceptic" world of Simon Singh, Ben Goldacre et al. Despite that, and the despite the fact that it focuses on neuroscience, it somehow seems very relevant to climate science, although I'm too tired to quite put a finger on it at the moment.

A recent set of articles in Perspectives on Psychological Science discussed inflated correlations between brain measures and behavioral criteria when measurement points (voxels) are deliberately selected to maximize criterion correlations (the target article was Vul, Harris, Winkielman, & Pashler, 2009). However, closer inspection reveals that this problem is only a special symptom of a broader methodological problem that characterizes all paradigmatic research, not just neuroscience. Researchers not only select voxels to
inflate effect size, they also select stimuli, task settings, favorable boundary conditions, dependent variables and independent variables, treatment levels, moderators, mediators, and multiple parameter settings in such a way that empirical phenomena become maximally visible and stable. In general, paradigms can be understood as conventional setups for producing idealized, inflated effects. Although the feasibility of representative designs is restricted, a viable remedy lies in a reorientation of paradigmatic research from the visibility of strong effect sizes to genuine validity and scientific scrutiny.

Article originally appeared on (http://www.bishop-hill.net/).
See website for complete article licensing information.