Bishop Hill Big questions
Apr 30, 2007 Climate: other Climate change is a big question. No, make that a huge question. A multi-trillion dollar question. The biggest question of our times and probably of any other times too. We are being asked to make devastating changes to our economies and to the way we live. Lives will be disrupted and ruined, but we are told that this we have to grit our teeth and deal with it. It's a necessary cost to bear in order to save the planet and a still worse fate.
This being the case, here are a couple of questions:
1. Shouldn't all the scientific research be replicated? I don't mean peer-reviewed - that's just a way of trying to cut out non-original work and any scientific "howlers". What I mean is take the raw data and turn it into the same results as were published in the original papers. This isn't done now because the climate scientists often don't archive their data, and don't release it when asked.
2. We should be able to see all the comments made by IPCC the reviewers of the IPCC reports so that the public can assess the firmness of the consensus. Complete agreement between 2500 scientists is simply not credible.
That's not too much to ask, is it? (Actually I'm sure it is).
Reader Comments (23)
Peer reviewers' reports are not generally made public-- is there any reason why an exception should be made here as opposed to, say, medical literature? At the end of the day, you trust editors' integrity or you don't, I guess. I've been a journal editor for many years and I've never seen any reason to doubt it. Peer reviewers are sometimes (rarely) a bit questionable, but each paper is seen by two, three or even more reviewers, they all get to see each other's comments upon revisions (papers are revised often before publication), and the editor oversees all. I don't think you need to be so suspicious of the system, seems to be pretty fair to me -- and I have no axe to grind.
You will of course know of Steve McIntyre's greivances over access to climate science data. The Climate Audit blog is chock full of examples of refusals to release data.
As an interested amateur I'm keen to see all sides to the argument - frankly I don't care about people's tone, their credentials or their prejudices. If they have something to say, I'll listen.
If someone is saying they can't reproduce reported results because of a lack of data then frankly I'm appalled. This would be scandalous in normal circumstances. For it to happen in something of such vital importance as climate science is simply unbelievable.
Jones et al 1990 is a case in point. McIntyre has had to resort to a request under the Environmental Information Act to try to get to identify the locations of the weather stations used. The response is here
http://www.cru.uea.ac.uk/cru/data/jonesetal1990/
This data is required in order to assess and reproduce the results. Why then, was it not released on request? I'm sure you don't think that this is acceptable.
There are some examples of failure to archive data here.
http://www.climateaudit.org/?p=1285#comment-99060
I realise that peer review is not normally public, but climate science is not normal. For a start, the IPCC report is a review of peer-reviewed science rather than publication of original work. Secondly, climate science it is far too important to be done in private. Why does peer review need to be done in private anyway? So as not to offend people's sensitivities perhaps. In climate science this kind of consideration must surely go out of the window. We're talking about a multi-trillion dollar question. Surely we're not going to keep it secret because someone might get upset by some criticism of their work?
I work for a journal that publishes original research in this area, papers that form some of the basis of the IPCC climate assessments. The IPCC assessments aren't pretending to be something that they aren't.
Obviously I'm not going to convince you because from what you say you have formed your views, and you will continue to pick holes. But I see things not from within, but from near enough to know that there are not consipracies, and that the vast majority of scientists are people who are interested in finding out what the data show them. Climate is a hugely politicised issue, it is easy to carp and criticise, but the people doing the research are honest.
There are plenty of websites and blogs that weigh up the scientific arguments: the best one I've come across is www.realclimate.org/ , and my own company, Nature Publshing Group, is going to launch a website on climate research soon (see www.nature.com/siteindex).
My advice is to trust one of these sites, not that I imagine you will be interested in taking it, based on what you've written above.
I have a view, yes. I'm sure most people do. I'm not dogmatic about it though, and I try to follow both sides of the argument. Consequently I read both Real Climate and Climate Audit. I find the latter somewhat more convincing, but I'm happy to be persuaded that I'm wrong.
This is besides the point though. McIntyre is clearly unable to get the data he needs to verify the paleoclimate papers he is interested in. This data should have been archived and hasn't. You don't specifically address the points I raised in my previous comment as to whether he should get it or not. You only say that you don't accept my view of McIntyre's actions. What do you mean by this? Do you think that scientists should be able to withhold data if they don't approve of those requesting it? How is that going to help us get to the truth?
You suggest that I take your advice and trust Real Climate or Nature.com. This is basically an argument from authority. Why shouldn't I listen to everyone and make my own mind up? McIntyre is clearly someone who is taken seriously enough to be referenced in the IPCC report and to be called as an expert witness before the US Congress. Mightn't he have something pertinent to say?
When you say "based on what you've written above" what do you mean? I can't see what I've written that has prompted this. The posting is a plea for openness in science. You're in favour of that aren't you?
I think Maxine meant to say that she sees more clearly than you, that as an outsider you don't really know what you are talking about - she sees it from the inside you see, and that you need to read real climate and nothing else. You see, climate science is highly politicised but she is not.
Personally, I am in the same position you are. Somewhat agnostic, suspicious of both sides of the argument. However, I am put off by the strident doomsday commentary and ruthless tactics (their willingness to launch into ad hominem attacks is particularly abhorent) of the proponents of anthropogenic catastrophic climate change, and for this reason rather sympathetic to the other side.
I have stopped reading Nature.
While there is obviously a considerable amount of archived climate data, there are also many deficiencies, especially with the Hockey Team. I've listed many defects with the various Hockey Team studies. As long as people like Maxine think the way she does, people like Phil Jones will be able to withhold not merely their data, but even the identification of their sites. Maxine's characterization of IPCC is much too polyannish - Hegerl et al who later published in Nature refused to supply even a list of sites to me as an IPCC reviewer; the sites were not published in the Nature SI. After over 20 emails stretching over nearly 2 years, I am still unable to identify all the sites used in this article. I've tried to resolve this cordially with Tom Crowley and had pleasant but unproductive exchanges with him. Maxine, unless this is resolved, you can expect another Materials Complaint.
I was threatened with expulsion as an IPCC reviewer by Susan Solomon for seeking data from authors. Her warning to me was that data evaluation was the province of the journals.
Nature refused to require Mann to provide full particulars of his method or even to require him to provide the results of his individual steps or his verification statistics, even after Nature knew that there were sufficient problems with MBH to require a Corrigendum (which was not externally peer reviewed by the way.)
I am particularly puzzled as to what "actions" of mine Maxine has a "different view" of. I've published a few articles, written a blog, corresponded with several journals and institutions, testified to the NAS Panel and a congressional committee - are these what she objects to? If so, on what grounds?
I assume she is referring to this correspondence
http://www.climateaudit.org/?p=782
MBH98 had 11 steps. Their latest step used proxies available from 1820 on and included instrumental series. It had high verification r2 scores which were published in a Nature figure. Our concern was over the early steps, especially the AD1400 step with only 22 proxies. We wanted to do statistical analysis of this step using Mann's actual results for this step - not the spliced results.
In any other field, supplying the results of the individual step would be routine. Mann refused. We filed a complaint with Nature. They provided a new SI (the provision of which was not peer reviewed). The new SI, while providing a considerable amount of information, once again failed to supply the results of the individual steps that we had originally requested.
On Aug 10, 2004, after publication of the Mann Corrigendum and SI, we re-iterated our request for the results of the individual steps, Nature took this under consideration and once again refused on the basis that they did not "normally" require this of authors and closed the book on the matter.
They also refused to require Mann to disclose source code despite evidence that his descriptions of his procedure were not accurate.
In her correspondence to me, Maxine Clark stated : "I don’t appreciate the tone of the comments you make to me personally, either, which are uncalled-for." This comment made no sense to me (or to other CA readers) as I had made no personal comments; all my comments were about the actions of the corporation. See http://www.climateaudit.org/?p=782
As to her view on access to data at IPCC, I've discussed my experience in which IPCC WG1 Chair Susan Solomon threatened to expel me as a reviewer for asking authors for data at http://www.climateaudit.org/?p=640. (One of the authors in question later publioshing in Nautre, still without identifying the location of the data.)
I'm also not a climate research specialist, so would not comment on those aspects. I do know, however, that IPCC is an assessment of the peer-reviewed scientific literature, among other things. "Just because 2500 scientists agree". Well, you would trust a pilot to fly a plane, rather than ask the guy sitting down the end row without qualifications, wouldn't you?
The reason I recommended the realclimate site is that it does not have a "line", it is an analysis of various articles. I don't very often read it myself, but it seems to me to provide a pretty informed view to people who aren't specialists, and who aren't at one of the polarised extremes that exists in this scientific area.
I'm not able to comment on what Prof M says about his papers or about what he has or hasn't requested in the way of data, as I have no knowledge of that and I don't like to speculate on what I don't know about. I do know about an email he once sent me, though, and the way it was phrased.
I think if I'd known that my "let's try to be sensible" comments were going to start all this, I wouldn't have made them in the first place. Will I ever learn? ;-)
There are other areas of science like climate research that are highly politicised and in which many people who don't work in the field and don't do the actual research feel qualifed to weigh in about what they think it all means or how unfair "the other side" is. Independent peer review provides a check and balance of someone's work that they cannot themselves have. Most scientists believe that the peer-review system works pretty well, on the whole (across all disciplines).
Thanks.
Ha Ha!
http://www.journals.uchicago.edu/JPE/datapolicy.html
Much of the hockey stick controversy could have been avoided if Nature had had similar policies. Note that data alone are not enough. There can be no doubt that the description of methods in the paper by Mann, Bradley and Hughes was insufficient to allow replication even if all the data had been provided. This was, among other things, because the "principal components" analysis provided was performed against the mean for a subset of the data not the whole data set and there were a series of steps to based on successively smaller numbers of proxies to allow the analysis to go further back in time. These would have been made crystal clear by archived code. Instead we had a series of successively closer approximations and revelations of how the analysis had been performed. Scientific controversy should be concentrated on the science not on establishing exactly what is was that someone did. In my opinion Nature should have data archiving polices like those of the major journals in economics.
Our personal views on Real Climate or Climate Audit are,of course, not really relevant to the thread. While you and I might disagree with Maxine, she will form her own opinions.
I think your second comment on Nature's policies are much more on the button. I'm intrigued by the idea of requiring submission of data and code to the journal itself pre-publication, rather than placing the data in a public archive (and presumably making code available on request). This strikes me as a much clearer and cleaner policy, which would also make life easier for the journal editors who would no longer have to chase recalcitrant authors.
Thanks for commenting again. I appreciate your taking the time to do so.
As I reiterated in my last comment, this posting was a plea for openness in science. Whether you and I disagree on Real Climate's "line", whether the tone of Steve McIntyre's correspondence is unreasonable or not, what people's motivations are; these are all issues which are irrelevant to the posting.
We should be able to agree that openness is vital to science. It shouldn't be controversial that scientific papers should be replicable. The question is, how can this best be achieved? To this end, what do you make of mikep's last comment?
I understand your comment about peer review, but I think that this misses the point. As I noted in the original posting, peer review can only eliminate non-original work and work that contains obvious "howlers". In these limited aims, I'm sure it is quite effective. But peer review doesn't attempt to replicate the science - this is something that has to be left to others to try at a later date. This being the case, we have to ask ourselves how to facilitate this work. The answer is, as I have said, to make available any materials that are required to replicate the science.
So strictly speaking, none of the results or graphs published by Nature can be used by US Government as pretty well all of it is based on unarchived data and methods.
There is a real risk to their reputation which they can only address through clear policies and maximum openness. Mikep's idea of submission of data and code direct to the journal would appear to be the best way of achieving this.
I've cited economic journal policies on many occasions. I sent the AER policy to Karl Ziemelis of NAture 3 years ago. In asking for archiving of data and methods in paleoclimate, this is merely asking for paleoclimate to adhere to a "best practices" policy already achieved in other disciplines.
Economists observed that provision of data and code reduces the cost of replication and verification dramatically.
Moreover not many peer-reviewers will analyse data separately - they might say that it looks as though there is something wrong with a particular derivation or estimation method, but they have neither the time or the incentive to re-work other people's submissions.
This leads on to the nature of the IPCC process. Maxine says "the IPCC process is a massive exercise in analysing each other's data and models.". But it does not seem to be like that at all. What is seems to be is a literature review where peer review is taken to mean that the research in question is fit for purpose. As Steve Macintyre has pointed out he, as one of the (2,500?) reviewers, was forbidden to ask for supporting data to analyse. Given the importance of the climate change questions and given the way IPCC works it is even more important than in other areas that data and code should be available for replication. The problem in this area compared to the experimental sciences is that it is not possible to replicate results by performing more experiments. The same is true in economics and perhaps that explains why the best economics journals have the policies about data and code that they do.