Big questions

Climate change is a big question. No, make that a huge question. A multi-trillion dollar question. The biggest question of our times and probably of any other times too. We are being asked to make devastating changes to our economies and to the way we live. Lives will be disrupted and ruined,  but we are told that this we have to grit our teeth and deal with it. It's a necessary cost to bear in order to save the planet and a still worse fate.

This being the case, here are a couple of questions:

1. Shouldn't all the scientific research be replicated? I don't mean peer-reviewed - that's just a way of trying to cut out non-original work and any scientific "howlers". What I mean is take the raw data and turn it into the same results as were published in the original papers. This isn't done now because the climate scientists often don't archive their data, and don't release it when asked.

2. We should be able to see all the comments made by IPCC the reviewers of the IPCC reports so that the public can assess the firmness of the consensus. Complete agreement between 2500 scientists is simply not credible.

That's not too much to ask, is it? (Actually I'm sure it is).

Comments

[Steve McIntyre]

The U.S. National Science Foundation does not require archive data or code; indeed, one of their officers, to whom I had copied a request, preemptively told Mann that he did not have to provide me with data that I had requested before Mann had even refused.

I've cited economic journal policies on many occasions. I sent the AER policy to Karl Ziemelis of NAture 3 years ago. In asking for archiving of data and methods in paleoclimate, this is merely asking for paleoclimate to adhere to a "best practices" policy already achieved in other disciplines.

Economists observed that provision of data and code reduces the cost of replication and verification dramatically.

[mikep]

There are some further points about peer review that would be worth exploring. I have no problem in accepting that peer review is generally fair (I've done it myself!). But it provides at best only limited assurance. There are plenty of examples in economics of papers which were initially rejected which were subsequently published elsewhere and became very influential. Similarly there are published peer-reviewed papers which were subsequently shown to be mistaken or inadequate. In economics at least it's the controversy in the journals which is as least as important as the initial peer review. Given that it is particularly important to allow controversy in the journals (and preferably the originating journals).

Moreover not many peer-reviewers will analyse data separately - they might say that it looks as though there is something wrong with a particular derivation or estimation method, but they have neither the time or the incentive to re-work other people's submissions.

This leads on to the nature of the IPCC process. Maxine says "the IPCC process is a massive exercise in analysing each other's data and models.". But it does not seem to be like that at all. What is seems to be is a literature review where peer review is taken to mean that the research in question is fit for purpose. As Steve Macintyre has pointed out he, as one of the (2,500?) reviewers, was forbidden to ask for supporting data to analyse. Given the importance of the climate change questions and given the way IPCC works it is even more important than in other areas that data and code should be available for replication. The problem in this area compared to the experimental sciences is that it is not possible to replicate results by performing more experiments. The same is true in economics and perhaps that explains why the best economics journals have the policies about data and code that they do.

[Not Maxine]

Maxine, was it something I said?