Bishop Hill IJoC - business as usual
Mar 17, 2011 Climate: other Journals Long-term readers may remember my efforts to get the International Journal of Climatology to adopt a sensible policy on data and materials - this was prompted by Steve McIntyre's attempts to extract information from the journal and one of its authors, Ben Santer.
At that point the journal had no policy, simply referring requesters to the author, and apparently happy to let the authors refuse if they wished. IJOC is a journal of the Royal Meterological Society, and the society's head, Paul Hardaker, was initially very favourable, with an undertaking to instigate a review. However, as months turned into years it became fairly clear that the society was caught between a rock and a hard place. If their policy was tough enough to ensure that data became disclosable then mainstream climatologists would not publish there. Climategate brought some confirmation of this, with the revelation of an email in which Santer and Jones discussed a boycott of the journal over a future data policy. Santer's words:
If the RMS is going to require authors to make ALL data available - raw data PLUS results from all intermediate calculations - I will not submit any further papers to RMS journals.
I have returned to Prof Hardaker every six months or so to inquire after the policy, and I discovered today that the policy has finally appeared on the journal's website.
The basic policy looks rather weak to me:
...a condition of publication in a Royal Meteorological Society journal is that authors are required, if requested, to make materials and data promptly available to readers where it is possible to do so under the restrictions of institutional or third party licensing agreements. Any restrictions on the availability of materials or information must be disclosed to the Editor at the time of submission...
Any data associated with the submitted article must be made available to Editors and peer-reviewers at the time of review, if requested by the Editor, in order to ensure a comprehensive peer-review process...
After publication, readers who encounter refusal by the authors to comply with this policy should contact the Editor-in-Chief of the journal. In cases where Editors are unable to resolve a complaint, the journal may refer the matter to the Society through its Chief Executive.
This in essence looks like the Nature policy - you don't need to submit data unless we ask (and who can forget Schneider telling McIntyre that he'd never received a request from a peer reviewer to examine data in 28 years as a journal editor). As I've said before, this simply stores up trouble for the journal. If authors refuse to disclose their data, the journal has few options - in fact in IJoC's case they only give themselves the option to "refer the matter to the society". I can hear the authors quaking at the thought of it.
THere is one caveat to all this - there is a separate policy on data papers, which the journal defines as papers about a data set.
Authors of such papers are required to deposit their data sets in a data centre that meets the criteria discussed above...
Data sets that are the basis of data papers will be subject to review. A sample of these data sufficient for the review process must be supplied with the submission of the paper. The reviewer is expected to comment on the data as if they were an integral part of the paper.
Data sets for data papers must include a descriptive ‘metadata’ section that provides the user with key information about the collection, preparation and use of the data set.
It strikes me that there is a lot of wriggle room here.
Reader Comments (11)
.......... this is good news for it again illustrates that climate scientists are not being open and transparent when it comes to publishing climate research. So too for climate journals who are unwilling to change the behaviour of climate scientists. Climate scientists remain an unreformed bunch.
This arms sceptics with a big question mark to lay against published climate research, "How many 'Nature Tricks' are out there, how many more are being planned?"
An additional observation:
Archival and accessibility of model code and data processing code is not required, being classed as meta-data with quite vague requriements. Presumably not even reviewers have the right to request this.
A previous column discussed why disclosure of code is part of the explication appropriate to data-processing-intensive papers. I don't see the reason for IJoC to exclude code; the description given in most papers to processing steps is far too generic. It's akin to describing a chemical experiment by "add a little bit of CH3COOH followed by some NaCl" -- without the details, it's not reproducible. Isn't that the purpose of archival and accessibility?
It says much about the area is what should be standard practice within science, the ability to access the data a research claims are made on, is so easily and readily avoided and worse that there is a real will to avoid it has well.
".......... this is good news"
How can you say that with a straight face is beyond me. Appalling, it all. This is anything but bad news, and science overall will suffer for these shenanigans and arrogant pricks.
Simply put: if an author is going to hide behind a weak policy, then his results must be discounted until such time as some other researcher can replicate his work without secrecy.
If you don't show your work then it's not a paper, it's a press release.
How in the name of Jove (climate god) is a circumstance that will transpire only after publication of a paper - i.e., people asking its underlying data - be a condition for publication of the paper?
This is beyond embarrasing.
I've held for a long time that Santer misspoke in his email. I've corrected his text as I believe it was intended, thus:
"♪It don't mean a thing ♪ if it ain't got that swing...♫" No replicability = no science.
In 'climate science', the data constantly changes. To attempt to reproduce a claimed result, one needs the exact same data (and code) set that was used for the published paper, not data from an archive that has been 'updated' by the paper's authors subsequent to publication. Since some of those who publish in this field are manifestly untrustworthy, all authors should be required to publish a hash code for the data in the paper itself. Someone who subsequently downloads archived data can then recalculate the hash and confirm that the data set is identical to that the authors claim to have used in their publication.
................. it is good news because we know that few bad apples have been allowed to spoil the entirety of climate science. Until the Team are picked clean climate science will remain rotten in peoples' minds.
QFT:
I think, in a very elemental sense, this is going to have to become the distinguishing feature of research over PR bluster in the future. Lack of ability to replicate, and lack of transparency in means and methodologies, are at the heart of every piece of junk science.
Any researcher who demonstrates a predisposition to behave like a junk scientist should be presumed to BE a junk scientist, their work disregarded and their funding suspended. We need a new age of ethical science, or moreover an urgent repair of the founding principles of science which have been allowed to erode away through neglect.