Data archiving

Steve McIntyre at Climate Audit has been trying to get hold of the dataset underlying a study by prominent IPCC author, Ben Santer. Santer himself has (rather snottily) told him that he can't have the data, while the journal in question, the International Journal of Climatology, has refused to help too. The editor, one Dr MacGregor, has said that they do not require authors to archive data as a condition of publication and that data can be obtained from the authors. This was rather cheeky, given that the authors had already refused to release a thing.

The International Journal of Climatology is a published by Wiley on behalf of the Royal Meteorological Society. I have now written to the Chief Executive of the RMS asking for a statement on the society's position on the issue

Dear Professor Hardaker

I read with great interest a recent article on the Climate Audit blog http://www.climateaudit.org/?p=4742, which reported on attempts to obtain copies of the datasets underlying a paper published in the International Journal of Climatology. According to the journal's editor, Dr MacGregor:

"It is not the policy of the International Journal of Climatology to require that data sets used in analyses be made available as a condition of publication. Rather if individuals are interested in the data on which papers are based then they are encouraged to communicate directly with the authors."

I found it frankly rather amazing that a journal would adopt a position of not requiring datasets to be availableas a condition of publication,given the importance of replication in the scientific process. It also seems somewhat unhelpfulof the editor to suggest a direct approach to the authors as a possible remedy, given that the authors in question had already turned down such a request. Mr McIntyre at Climate Audit has subsequently made unfavourable comparisons between the policies of IJC and the Royal Society's Phil Trans B which has adopted and enforced a policy on data archiving (see http://www.climateaudit.org/?p=4754), and given that the paper in question is now essentially unreplicatable, I find it hard to disagree with this position.

Since this is a matter of pressing public concern, I was wondering if you would care to issue a statement on the Society's position on the importance of replication and the availability of data used in its publications, both in general terms and in relation to the paper in question. I think it is important for the Society to make a stand on the credibility of the papers that are published under its name. This would be an excellent subject for a posting on your blog.

I have published this letter on my own website at bishophill.squarespace.com and will of course link to or publish any response you give.

Kind regards

My feeling on this issue is that the Society has probably never concerned itself with nitty-gritty issues like data archiving, but that they could and should. It will be interesting to see what Professor Hardaker has to say.

(Professor Hardaker's blog is here).

Comments

[per]

well, okay.

but you need to careful of the word-smithing here.

the relevant data are publically available, and are appropriately described.
what isn't provided, and what steve wants, is santer's workings, including a copy of the data set used, and code needed to take it through the calculations.

Point being that unless you work to the specification of working to that degree of auditability (which is a requirement in many industries), then you have to re-do the whole analysis, which would take time. The archiving of your workings is not a commonly-accepted standard.

per

[Bishop Hill]

Yes, the code is easily forgotten, and I confess that this is what I've done here. I hope that Professor Hardaker will take my letter in the spirit in which it was written and address the wider question of ability of third parties to replicate the study as well as the question of the data.

[James P]

Maybe I'm naive, but why should anyone take seriously any paper where the means to reproduce the study/experiment is not readily available? This is a contentious topic (otherwise why write about it?) and the only conclusion I can draw from secrecy over data or method is that there is something to hide.

BTW, here in the Isle of Wight, which is normally a very temperate spot, being small and surrounded by water, we have just had -5.6degC air temperature, which is as cold as I can remember. If this is global warming continues, it looks like we all may freeze.. :-)

[Perry]

From http://en.wikipedia.org/wiki/Scientific_Method

"Among other facets shared by the various fields of inquiry is the conviction that the process be objective to reduce a biased interpretation of the results. Another basic expectation is to document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, thereby allowing other researchers the opportunity to verify results by attempting to reproduce them. This practice, called full disclosure, also allows statistical measures of the reliability of these data to be established."

This year being the bicentenary of the birth of one C. Darwin Esq., begs the question of would he have been recognised as the scientist we know today, if his book had not set out the rationale, all his evidence, for his conclusions? So it is with Santer. His conclusions are worthless without verification by others. The fact that he is behaving like the little boy who takes his ball home, so the others cannot play, only serves to underline his fear: that he suspects he may be W R O N G.

[James P]

I found this today, from a surprising source:

“I know that most men, including those at ease with problems of the greatest complexity, can seldom accept even the simplest and most obvious truth if it be such as would oblige them to admit the falsity of conclusions which they delighted in explaining to colleagues, which they have proudly taught to others, and which they have woven, thread by thread, into the fabric of their lives.”
.
.
.
.
.
.
.
(Leo Tolstoy)