Praising post-publication peer review

Richard Smith, former editor of the British Medical Journal and regular critic of the pre-publication peer review process, writes on the subject of post-publication peer review.

But more important than...formal types of peer review is the informal, the thousands of comments, decisions, and actions from the many that lead to a sorting of studies. I may hear a study presented or read a paper and be impressed. Others in the audience or other readers might also be impressed. We talk to friends about it. We email colleagues. We put it on listserves. Some of the recipients are impressed and start their own cascade. Others are less impressed and see problems. Perhaps a statistician attracted by the clamour reads the clinical article and sees important flaws that she shares with colleagues. Somebody might incorporate the study into a lecture, a review, or a grant application. And so a study might attract increasing attention and assume a prominent place, or it might fade as its receives more attention and more problems are noticed.

Many studies, in contrast, attract no attention—usually, but not always, rightly.

Physics World comes over all sceptic

Physics World, until now a bastion of conventional thinking on AGW, has come over all sceptic, with the Quanta column featuring all manner of AGW unfriendly stuff.

Nature Climate Change launches

Nature's long-awaited climate change journal has finally launched. It looks as though it's all free in this first issue, so why not take a look?

This article about openness in climate change research was interesting, with several familiar faces interviewed. There is something about the tone of the piece that makes me uncomfortable though - perhaps a slightly promotional feel?

Quote of the day

The anonymous peer review process is the enemy of scientific creativity…peer reviewers go for orthodoxy.

Professor James Black

Quoted in Donald Gillies' submission to the House of Commons Science and Technology COmmittee. H/T Judith Curry.

SciTech peer review inquiry

The House of Commons Science and Technology Committee have started to publish the submissions of evidence on their website.

A number of familiar names are there, and I'll try to read these when I get a chance:

• Philip Campbell, the editor of Nature who had to resign from the Russell inquiry after prejudging the findings (not to mention his conflict of interest)
• Richard Horton, the editor of The Lancet, who replaced Campbell and whose advice was ignored where Russell found in convenient to do so
• Michael Kelly, of the Oxburgh panel, whose observations on the indequacies of CRU's work was not reported by Oxburgh
• Nic Lewis, of O'Donnell et al fame
• Prominent sceptics, McLean, de Freitas and Carter

There are also two from UEA and one each from the big learned societies, including the Royal Society.

Update on Mar 17, 2011 by Bishop Hill

Richard Horton's evidence includes this:

Peer review is a central issue in many scientific controversies and disputes today.  Take climate change. In the Times Higher Education , last year, Andrew Montford, author of The Hockey Stick Illusion: Climategate and the Corruption of Science (1), argued that events at the Climatic Research Centre (UK) at the University of East Anglia (CRU) had far-reaching implications for the world of scientific peer review and publishing (2). His charge sheet was sharp and precise: that scientists undermined the peer-review process. Implicit in Montford's argument is that peer review is critical to the process of – and thereby public trust in – science. Writing in The Guardian , George Monbiot put it this way: "science happens to be [a] closed world with one of the most effective forms of self-regulation: the peer review process."(3). But for many of us who do peer review, this "most effective" form of self-regulation is often misunderstood and misrepresented.

IJoC - business as usual

Long-term readers may remember my efforts to get the International Journal of Climatology to adopt a sensible policy on data and materials - this was prompted by Steve McIntyre's attempts to extract information from the journal and one of its authors, Ben Santer.

At that point the journal had no policy, simply referring requesters to the author, and apparently happy to let the authors refuse if they wished. IJOC is a journal of the Royal Meterological Society, and the society's head, Paul Hardaker, was initially very favourable, with an undertaking to instigate a review. However, as months turned into years it became fairly clear that the society was caught between a rock and a hard place. If their policy was tough enough to ensure that data became disclosable then mainstream climatologists would not publish there. Climategate brought some confirmation of this, with the revelation of an email in which Santer and Jones discussed a boycott of the journal over a future data policy. Santer's words:

If the RMS is going to require authors to make ALL data available - raw data PLUS results from all intermediate calculations - I will not submit any further papers to RMS journals.

Click to read more ...

How to publish a comment

H/T to DI for this article by Rick Trebino, a physicist from Georgia Tech.

...in this article, I’ll share with you my recent experience publishing a Comment, so you can, too. There are just a few simple steps:

1. Read a paper that has a mistake in it.

2. Write and submit a Comment, politely correcting the mistake.

3. Enjoy your Comment in print along with the authors’ equally polite Reply, basking in the joy of having participated in the glorious scientific process and of the new friends you’ve made—

the authors whose research you’ve greatly assisted.

Ha ha! You didn’t really believe that, did you?

Read the whole thing.

Update on Mar 11, 2011 by Bishop Hill

Some users were reporting a virus at the Scribd document linked. I've changed to an alternative site (H/T Doug Keenan) but this is not responding at the moment.

Nature materials policy

Shub Niggurath has an in-depth look at Nature's materials policy - a must-read.

(Also posted at WUWT)

Science policy on data and code

This from reader Lance:

The Feb 11 Science magazine (v331, p649) states a new journal policy--they will now require that authors archive not only their data on a website (copy to be held at Science) but also their computer codes!

Think of the time and exasperation that Steve McI could have been spared had the journal that published the Hockey Stick had this policy in place.

On the other hand, the world would have lost the "rattlingly good" Hockey Stick Illusion as well.

I almost think that Science has been listening to M&M, WUWT, and Your Grace.

The new policy doesn't appear to be online. It's certainly welcome.

Liz Wager on conflicted peer review

There has been an interesting discussion in the Steig thread about whether Eric Steig should have been invited to be one of the reviewers of the O'Donnell paper or not. On the one had there is the fact that Steig, being the subject of the critique, had a conflict of interest. On the other, he would have been the person best able to point out possible flaws in the O'Donnell paper. Opinion among commenters appeared divided. With this in mind I wrote to Liz Wager at the Committee on Publication Ethics - an advisory body for scientific journals - to ask for her thoughts. Here they are:

Should an author whose work is the subject of a criticism in a submitted manuscript be among the invited peer reviewers of the manuscript?

COPE (the Committee on Publication Ethics) doesn't issue general guidance or proscriptions on how peer review should be done but we do mention criticisms in our Code of Conduct for editors, namely

"Cogent criticisms of published work should be published unless editors have convincing reasons why they cannot be. Authors of criticised material should be given the opportunity to respond.

Studies that challenge previous work published in the journal should be given an especially sympathetic hearing."

In developing this guidance, we had in mind letters to the editor rather than new analyses/papers but recognise that practices differ in different areas (apparently maths journals never print correspondence so if you want to criticise another person's work you have to write a new paper).  So COPE says that the authors should be given the opportunity to respond to specific criticism of their work, but we do not provide guidance about whether they should peer review papers criticising their research.  We leave that up to the editor.

Liz Wager, Chair COPE

Peer review isn't working

I'm grateful to reader Steve for pointing me to this article by Carl Phillips, an epidemiologist, who is looking at the efficacy of peer review. The whole article is worth a look, but here are some choice quotes:

Do the reviewers ever correct errors in the data or data collection?  They cannot – they never even see the data or learn what the data collection methods were.  Do they correct errors in calculation or choices of statistical analysis?  They cannot.  They never even know what calculations were done or what statistics were considered.  Think about what you read when you see the final published paper.  That is all the reviewers and editors ever see too.  (Note I have always tried to go the extra mile when submitting papers, to make this system work by posting the data somewhere and offering to show someone the details of any analytic method that is not fully explained.  This behavior is rare to the point that I cannot name anyone else, offhand, who does it.)

Does this mean that if you just make up the data, peer review will almost certainly fail to detect the subterfuge?  Correct.

Does this mean that if you cherrypick your statistical analyses to exaggerate your results, that peer review will not be able to detect it?  Correct.

 But it serves just fine for justifying the uprooting of the economy.

SciTech committee to investigate peer review

This could be interesting:

The Committee has today launched an inquiry into peer review. The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.

The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:

1. the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public;
2. measures to strengthen peer review;
3. the value and use of peer reviewed science on advancing and testing scientific knowledge;
4. the value and use of peer reviewed science in informing public debate;
5. the extent to which peer review varies between scientific disciplines and between countries across the world;
6. the processes by which reviewers with the requisite skills and knowledge are identified,  in particular as the volume of multi-disciplinary research increases;
7. the impact of IT and greater use of online resources on the peer review process; and
8. possible alternatives to peer review.

The Committee welcomes submissions from scientists whose material has been peer reviewed, those who commission peer reviews and those who carry out peer review.

The Committee invites all written submissions on any of these issues by Thursday 10 March 2011.

New climate journal?

This new journal looks as though it was set up specifically with climate sceptics in mind.

Open access and gatekeeping

Nature is to start up an open-access scientific journal. The new journal, to be called Scientific Reports, will cover biology, chemistry, the earth sciences and physics.

The story, in the Times Higher Education Supplement, concentrates on the implications for subscription-based journals, but it is interesting also to consider whether this will have any effect on attempts to keep sceptics out of the scientific literature.

Like the Public Library of Science's PLoS ONE journal, Scientific Reports will be entirely open access and will publish every submission deemed by a faster peer-review process to be technologically sound - including those reporting useful negative results.

One wonders if a "faster" peer-review process, 88 pages of peer review correspondence can simply be replaced with the word "No".

Toronto Sun on Climate Files

Lorrie Goldstein of the Toronto Sun asks if maybe climate science shouldn't be just a bit more open, and citing Fred Pearce's The Climate Files as evidence. The tone of the article is interesting, with Goldstein noting that Pearce is not a "denier", but pointing out his criticisms of the climatology community's failure to check its findings.

As well as taking pot shots at climatology peer review, he also has things to say about the Climategate inquiries:

Simply having panels of sympathetic academics (or politicians) take a cursory look at the work of climate scientists and pronounce it sound — what happened following Climategate — doesn’t cut it.