Whether to trust statistics

Betsey Stevenson and Justin Wolfers, writing at Bloomberg's website, consider some rules of thumb for helping the layman decide whether they should trust someone's statistical analysis or not. Here's the first of them:

Focus on how robust a finding is, meaning that different ways of looking at the evidence point to the same conclusion. Do the same patterns repeat in many data sets, in different countries, industries or eras? Are the findings fragile, changing as one makes small changes in how phenomena are measured, and do the results depend on whether particularly influential observations are included? Thanks to Moore’s Law of increasing computing power, it has never been easier or cheaper to assess, test and retest an interesting finding. If the author hasn’t made a convincing case, then don’t be convinced.

It's hard not to recall the case of the Hockey Stick and its reliance on the bristlecones. And all the other paleoclimate studies that are said to support the Mannian stick, and which rely on bristlecones too.

Comments

[HaroldW]

I second Nullius in Verba's call for goodwill (8:52 AM). It strikes me that the tone here has been hostile to Jim. I've had productive exchanges with him at his blog. We certainly don't agree on everything, but what a drab world it would be if we all did.

Jim has made, is making, a serious effort to try to evaluate quantitatively the reliability of tree-ring reconstructions of temperature. And he's doing it in what I consider to be a correct way, starting with applying the current techniques to toy problems, and seeing how well they work in an environment free of many of the confounding influences found in the real world samples. [Rather than an axiomatic assumption that a plausible technique always generates a correct temperature value plus simple noise.] I find what he wrote to be convincing, and might well fundamentally change the field. We should all be appreciative of his efforts, which have been met with some circle-the-wagons reactions. I'd encourage people to read his blog entries (begin here).

Understand that Jim comes to this discussion from a different perspective. It doesn't mean, though, that he necessarily shares the views of any other person, in particular at RC. There is a diversity of opinion even among those who believe AGW to be a serious threat; considering "consensus-holders" as monolithic is as unhelpful as those who paint all contrarian view as "GW deniers".

I don't care for the "two sides" approach to these discussions; it oversimplifies and fosters polarization. But if one takes such a view, one should be willing to admit that there are some on "your side" (whichever side that is) who over-egg the pudding. It doesn't invalidate your perspective -- nor the other side's. Avoid the personal, concentrate on the factual arguments, don't attribute opinions by association...in short, good will.