Bishop Hill Liz Wager on conflicted peer review
Feb 10, 2011 Climate: Surface Journals There has been an interesting discussion in the Steig thread about whether Eric Steig should have been invited to be one of the reviewers of the O'Donnell paper or not. On the one had there is the fact that Steig, being the subject of the critique, had a conflict of interest. On the other, he would have been the person best able to point out possible flaws in the O'Donnell paper. Opinion among commenters appeared divided. With this in mind I wrote to Liz Wager at the Committee on Publication Ethics - an advisory body for scientific journals - to ask for her thoughts. Here they are:
Should an author whose work is the subject of a criticism in a submitted manuscript be among the invited peer reviewers of the manuscript?
COPE (the Committee on Publication Ethics) doesn't issue general guidance or proscriptions on how peer review should be done but we do mention criticisms in our Code of Conduct for editors, namely
"Cogent criticisms of published work should be published unless editors have convincing reasons why they cannot be. Authors of criticised material should be given the opportunity to respond.
Studies that challenge previous work published in the journal should be given an especially sympathetic hearing."
In developing this guidance, we had in mind letters to the editor rather than new analyses/papers but recognise that practices differ in different areas (apparently maths journals never print correspondence so if you want to criticise another person's work you have to write a new paper). So COPE says that the authors should be given the opportunity to respond to specific criticism of their work, but we do not provide guidance about whether they should peer review papers criticising their research. We leave that up to the editor.
Liz Wager, Chair COPE
Reader Comments (106)
That Steig is, perhaps, the only person who would be familiar enough (presumably) with what he did for his paper probably means that he has to be involved. It also tells us a lot about the nature of Climate Science - otherwise there would be 4000 IPCC scientists all available to be asked to take on the task.
O'Donnell's complaint seems to recognise what was going on, in part, and appears to accept it but objects to the way it was handled as he understands the train of events. If his understanding is correct he is right to complain. If it is not it should be easy to explain the misunderstanding.
I'm off to buy popcorn.
Surely the issue here isn't whether eric should have been a reviewer or not but the lengths his reviews took. Is turning an 8 page paper in to 88 pages what passes for appropriate reviewing?
Secondly there is also the issue of integrity, The Team should have known that they would have exposed themselves to challenges to their integrity by reviewing the paper with stein. Does honesty and integrity mean nothing in climate science?
Thirdly, the Team should have known that by getting involved that they would tgen face accusations that their activity only confirms what many accused them of underminig the peer review process after sunlight exposed their activities in the past.
It just seems common sense to me that it was a very poor choice for Steig to get involved, which was only compounded by his continued denials at RC that he was reviewer A. All Strig has ended up doing is further exposing the games The Team gets up to when anything challenges their position.
Regards
Mailmab
Mailman
I am amused to see you refer to Eric as "Stein", "Strig" and "Steig". And to yourself as "Mailmab". :-)
I wonder if the conclusion from all this may be that it was not inappropriate to invite ES to review but that the editors should have shown a greater inclination to disregard his critique. (I should point out that I haven't read the peer review docs, so there is a level of surmise in my comment here).
Bish
And there was me thinking he was the stig!
Steig - old German word for steep path.
With a handle like that it is little wonder Eric needed the rest of the Team to help him roll 88 pages of comments up that particular peer-review hill.
No one disputes that Steig should have had some form of right to reply: the issue from my perspective is that Steig was an anonymous reviewer.
What has followed is the kind of car crash that "conflict of interest" considerations are meant to avoid.
Indeed on the issues around peer reviewed work is the way its actual done in practice varies. In one area a reviewer could not be the person whose work was being critiqued in other it would be expected they would be. Worse in can even varies with a subject depend on the Journal, there is no uniform standard, perhaps there should be , but there is a ‘ethical lookout’ that suggest that conflict of interest should be graded against.
What comes across in the reviewers comments is that Reviewer A is far more critical than the other two which on the face it may be fair enough. However the real difference is that reviewer A’s suggestions for change push the paper to make changes that would make it more supportive of Steig09 and not necessarily in interest of improving the research on its own merits. And that is where the conflict of interest comes in.
That the editor had to bring in forth reviewer because of the actions of reviewer A, suggest that they became aware that reviewer A interests where not in judging this article but in finding ways in which to use it to support the Steig09 one. If you did not known who reviewer A was, reading their notes alone would flag up the strong possibility that they had a strong relationship to Steig09 that was colour their views.
Has it is we known it was Steig and so we can better understand way reviewer A acted in the way they did.
It is quite clear that Eric Steig was operating beyond his competence in both the paper he had published and the critique of it.
He was reliant on others to get his paper published and for the substantial comments he offered up as reviewer.
Steig never operated as an author responding to O'Donnell he was a blocking reviewer. Steig understood that role perfectly
Bish,
I'm typing my responses on an iPhone, hence the shoddy keyboardsmanship :)
Regards
MailmaN :)
This post and the Bishop's comment on it align with what I was arguing back in the Steig thread. As a scientist who does quite a lot of publishing and refereeing (in another area), I would say that many's people view in my field would be that there's nothing wrong with an editor picking a potentially negatively biased referee - so long as that referee's report then gets taken with a large grain of salt. In this case, that referee got deferred to for too long in my opinion (though in the end the paper did not get too badly mangled as a result).
woodentop
Ah, our learned legal expert has made an excellent point. Put into his world, what happened is Steig smashed into O'Donnell's car, O'Donnell files suite and Steig is put on the jury.
Steig clearly had the right to give his side of the story, but from the witness box and not the jury box.
Hopefully, I got it right, woodentop
j
It would appear to me 88 pages of comment for a eight page paper is a very large deferral, and keeping the paper out of print for a year as a bad mangling.
I find that the editor blew it, to use the vernacular.
I have no quarrel that Steig should have had an opportunity to response, and that his response, given a review by the same reviewers as the main paper, should have been published with it. Furthermore, the author of the main paper should have been aware of it all.
but one of the main planks that is emerging is that Steig admits to not being a statistician...so why does that qualify him as a person to comment on a statistical exercise that shows that he is inept?
I agree with j. In my experience it is not unusual for the editor to send a critical paper to the criticisee for review. He has to keep in mind the potential bias of that referee, and it's clear from reading the correspondence now posted that the editor did so in this case. For example he made it clear that he did not insist on them changing the paper in response to all of the comments of Rev A. I think the editor acted very well in this case (Don Pablo, please read the letters from the editor to Ryan), and the paper was published.
The mess resulted partly from Steig admitting that he was a reviewer and then pretending that he wasn't (for example saying asking for a copy of the paper on Jeff's blog). Plus of course subsequently criticising the paper for including things that he himself had suggested.
Has he now admitted to being Rev A, or does everybody just assume that?
Being a clown, I'm not involved in scientific publishing. However, Mrs. Clown is indeed a managing editor of a couple of peer-reviewed journals in environmental health sciences. It is her considered opinion that one should not select, as a reviewer, any of the authors of the original paper being criticised in the submitted article. Peer review should be performed by competent, independent, and fair third parties - provided, of course, that the editor can find a sufficient number of reviewers who are competent, independent, and fair-minded. (It's not at all easy to do so, even in areas of science devoid of intense controversy.)
However, Mrs. Clown does feel that the authors of the criticised paper should be granted the right to view the new article pre-publication and to respond in the same issue of the journal with their rebuttals. It is a mystery why JoC didn't proceed in this manner.
According to Sturgeon's Law, ninety-five percent of everything is crap. Peer review is nothing more than a crude filter devised to shake out the blatant crap before it gets into print. When it is misused to suppress valid criticisms all of science suffers.
I find some of the commentary on Steig provocative.
Steig abused the privileges of being a reviewer by:
- Spinning out the review process to delay publication by O’Donnell et al.
- Causing O’Donnell et al. to alter their methodology and then critiquing the altered approach post publication
I believe O’Donnell himself has described the latter action as duplicitous. Which it is.
I do not for one instant believe that Steig ‘forgot’ that he caused the change in approach by O’Donnell et al.
There should be no apologists for Steig. End of story.
The editor’s role is another matter, but there is a danger of agreement that J Climate did not in fact behave badly somehow softening the perception of Steig’s behaviour.
Can we expect a new peer reviewed paper, discussing editorial policies, and peer review?
GP's point in the first comment above seems to me to be the killer. There are supposedly thousands of climate scientists out there who all agree on CAGW. Why couldn't one of them have done the peer review?
It's because the number of climate scientists who are also competent statisticians is roughly zero.
PaulM
(Don Pablo, please read the letters from the editor to Ryan)
Reference?? I have not seen the letters to Ryan, nor do I know where they are.
and the paper was published.
After about a year's delay, if I am not mistaken. I am not sure that the letters to Ryan are germane as they are merely words written either in email or on paper. The actual happenings suggest an apparently different course of events. That is why this deserves closer scrutiny. And as for Ryan's gentle comments regarding what the editor did, remember that he has to continue to have a professional relationship with the editor and the journal. I believe Ryan is being diplomatic, which is reasonable and wise.
The mess resulted partly from Steig admitting that he was a reviewer and then pretending that he wasn't
Which in my book, should not have happened in the first place. This is why "conflict of interest" exists, to prevent such "accidents".
I am not sure what you mean by "review". In my lexicon the review of a scientific paper is a objective review of a scientific work by those working and knowledgeable in a field. Such a review can keep a paper from being published, or insists on changes. It is a jury system. That is why it is called peer review.
On the other hand, I use the word "comment" to refer to permitting the author of the original paper the opportunity to speak to the issues.
I stick to my comment that Steig clearly had the right to give his side of the story, but from the witness box and not the jury box. None of this would have happened if the editor had kept that in mind, particularly with regard to a critical paper that basically tears apart previously published work.
I shall have to read about what COPE does manage to say, because not saying anything about offering people a way to conceal their identity so they can go on the attack seems an obvious area of ethical concern. One rather wonders what more urgent issues have pushed this one to the back of the queue.
Feb 10, 2011 at 2:53 PM | Unregistered CommenterDon Pablo de la Sierra
Don Pablo - Best analogy EVER and so spot on that it should have it's own post. It is exactly what happened.
Maybe it is time to adopt open peer-review across science. Open dialogue and procedural transparency should become the norm in climate science. That to me seems the only way to regulate this entire process.
I do not know the details of the journal's editor and his role in the affair. However, I am surprised that Steig was allowed to be a reviewer. He should have been able to argue his case, but not as an anonymous reviewer. Also, Steig was clearly guilty of an OTT response and delaying tactics and the editor should not have permitted that.
The Climategate affair alerted us to the sometimes cosy, sometimes threatening relationship between some climate scientists and journal editors. It makes me wonder about this case, but as I said, I don't know all the details.
@Alexej Buergin
He has:
realclimate
I have published in both physics and finance journals and I would EXPECT that if I was to write a paper that criticised the work of another, that that paper be reviewed by that person. After all the purpose of the editor is to determine whether the paper is valid and the best person to criticise it is the person being criticised, especially when the method underpinning the work is extremely technical and may not be widely understood.
This is what happened in this case. However, any competent editor KNOWS that there is a conflict here and so will choose at least 2 other reviewers so that the first reviewer does not have the ability to unilaterally veto the paper. This is what happened in this case.
Fred Bloggs - and others - are still (deliberately?) missing the point. So here's a repeat of my earlier comment:
I find some of the commentary on Steig provocative.
Steig abused the privileges of being a reviewer by:
- Spinning out the review process to delay publication by O’Donnell et al.
- Causing O’Donnell et al. to alter their methodology and then critiquing the altered approach post publication
I believe O’Donnell himself has described the latter action as duplicitous. Which it is.
I do not for one instant believe that Steig ‘forgot’ that he caused the change in approach by O’Donnell et al.
There should be no apologists for Steig. End of story.
The editor’s role is another matter, but there is a danger of agreement that J Climate did not in fact behave badly somehow softening the perception of Steig’s behaviour.
The editor’s role is another matter, but there is a danger of agreement that J Climate did not in fact behave badly somehow softening the perception of Steig’s behaviour.
Or distracting from it.
There is nothing wrong with Steig reviewing the O'Donnell et. al. manuscript but he should not have been an anonymous reviewer.
IIRC O'Donnell et. al. first attempted to write a letter to Nature but it was rejected, Steig suggested submitting a paper which was rejected by Nature. One has to wonder if this was because of the front-cover prominence which Nature had given Steig's work. O'Donnell et. al . then had to publish in another journal.
Since the main challenge to Steig's findings centered around statistical methodology I would have thought that one or more of the other reviewers should have been a statistician (maybe this was the case)
Steig could have made a public review and this would also have afforded him the opportunity to retract his paper if it was found to be terminally flawed. I don't think this is necessary as O'Donnell et. al are clear that they wanted to improve Steig's work not show him to be incompetent.
AFAIK the 88 pages of comments were not all Steig's. It is conceivable that the length of his review was increased because he wanted to know other (statisticians?) comments vie the editor to gauge if he'd made a serious statistical error. Such things would weigh heavily on his mind, I imagine.
Of course the duplicity in setting up O'Donnell for open criticism by using peer review comments to force a modification of the statistical methodology seems to be very underhand.
The papers in question are largely statistical not climatological. If one were to find/develop a new statistical technique to handle the data which showed that the whole of Antarctica was warming, what would it mean for the AGW debate? Just that the data tell you inconclusive/opposite things depending on how you do the stats.
Of course there is a problem with Steig being an anonymous reviewer of work which rebuts his own. But there is a bigger problem, shown very clearly in the Climategate emails. Which is that the 'hockey team' are indeed a team, and will 'protect their own'. I think we would have seen a very similar convoluted review process whoever reviewed this paper, and that Steig would have been consulted anyway privately by other reviewers.
timheyes
There is nothing wrong with Steig reviewing the O'Donnell et. al. manuscript but he should not have been an anonymous reviewer.
I think we all agree with that. I think the issue is in lexicon or semantics. I totally agree Steig should have read the paper and commented on it openly. A reviewer has much more power than that, and is almost always done anonymously. This is the same theory used in keeping the names of a jury anonymous in the legal system. And I might add, the same theory used for the secret ballot.
That is why I used the term COMMENT. Reviewers are voting. They are the jury or electorate.
@DPdlS - I wasn't really drawing a legal parallel, though yours is illustrative of what's happened here.
Suffice it to say that potential conflicts of interest abound in professional life, and one was must always be on guard to avoid even the appearance of them lest one's motives are called into question. It's ethics 101.
An open dispute with a critic of your work is to be expected; to hide behind a cloak of anonymity and snipe from a position of relative power is not. If the O'Donnell et al paper had been rejected as a result of this extended review process, would any of these shenanigans have come to light?
@ Don Pablo. I wasn't arguing with you - just sticking my 2 cents in.
I think an important issue is being overlooked. Before Climategate the Team would have got away with this. Climategate allowed the public to see what they were up to in the peer review process. They are being exposed for what they really are!
And Steig was well aware of the potential for a conflict of interest problem:
He knew there would be "abuse" if it leaked out that he had been asked to review a paper criticising his work. And yet ...... he went ahead and agreed to review anyway;
... he tried to keep it quiet;
... he then dug himself a hole by tearing O'Donnell to pieces for using a different statistical method which (depending on whose version you read) Steig had himself suggested or O'Donnell had used because Steig wasn't happy with the original which Steig then said he preferred to the one used.
And he's p***ed off that O'Donnell decided all bets were off on the confidentiality front. What is it with these guys?
The more I read the more it seems that climate science is irrelevant. The name of the game is wrong-footing anyone who disagrees with you.
A bit of inter-university rivalry is one thing but are we not all at least supposed to be aiming for the same goal?
(Don't worry; I'm not so naive that you need to actually answer that last question!)
RE: TimHeyes
"AFAIK the 88 pages of comments were not all Steig's. It is conceivable that the length of his review was increased because he wanted to know other (statisticians?) comments vie the editor to gauge if he'd made a serious statistical error. Such things would weigh heavily on his mind, I imagine."
1. We do not know if Steig wrote all the comment himself. Many observers have speculated this was a Team effort, including JeffID.
2. Not all the 88 pages were written by Steig - some of those pages are resulting response by the author's to Steig. Nonetheless the review traffic created by Steig as Reviewer A is excessive given the paper is just 8 pages long.
3. Steig may well not be fully competent to review the statistics. In that case he should have declined to review the paper, even for no other reason. I would have declined any paper I was aked to review on that basis, or I would have reviewed in general terms and commented on the statistics that I was not qualified to review this - as Reviewer B did:
Reviewer B: "I am not conversant with the statistical nuances of the analyses by Steig et al. and the approach adopted here, so trust that Eric Steig or Michael Mann will provide that needed expertise!".
However, note something odd about that anonymous comment - Reviewer B is suggesting that Steig or Mann will provide the expertise to critique the statistics in O'Donnell. This raise 2 questions: is reviewer B part of the Team and/or did Reviewer B know who Reviewer A was?
4. To pass the paper to other parties for additional review and then pass the comments back to the editor as your own review would, in my opinion, be completely unethical. We do not know if this was done (see 1 above).
5. To tell the editor that you are going to pass the paper to someone else because you do not have competence on the statistics should not normally be allowed - the editor would want to know who, and if you have a recommendation the editor should normally contact them as an extra or alterante reviewer directly.
Looks like Andrew Revkin has nothing to say.
Incredible.
Don Pablo, I agree. Tim Heyes statement should read:
"There is nothing wrong with Steig commenting on the O'Donnell et. al. manuscript but he should not have been an anonymous reviewer."
The first post that brought me to Bishop Hill in the frst place was "Caspar and the Jesusu Paper"; in which we saw that Steve McIntyre was the anonymous reviewer of the paper submitted to Climatic Change.
Whilst I have infered from that article that it is rather unusual for a paper which proports to repudiate and existing paper to be reviewed by one of the repudiated, it is not unknown.
Therefore, I have to say that the choice of Steig as the reviewer is in a similar terrritory as Steves choice back in 2005.
KJ
Knockjohn
Which is to miss the point. Why are so many people doing this?
Please see my comments above at 4:54 and 5:00pm. I have already re-posted it once on this thread, so I won't copy it here again.
BBD, you are indeed correct regarding the behaviour of Steig as reviewer. I did not intend to give the impression that the choice of reviewer by J Climate should excuse the actions undertaken by the individual in the post.
I merely wished to bring to the fore, that for a journal to select a far from disinterested person is not of itself unheard of in this arena.
KJ
KnockJohn - Amman and Wahl's CC paper that McIntyre was asked to review purported to replicate Mann's Hockey Stick, which McIntyre had previously criticised. That's a completely different situation from having the author of a paper anonymously reviewing (and blocking?) a submission which is directly critical of that author's paper.
If Steig had reviewed a paper which supported O'Donnell et al, your assertion would be on firmer ground.
Incidentally KnockJohn, can you summarise for readers the similarities in behaviour between Steve McI and Steig in their role as reviewers?
Because that's the central issue here, not whether or not J Climate should have allowed Steig to participate.
Did Steve McI try to sabotage the review process? Did he try to sabotage Amman & Wahl?
Let's not lose sight of the real issue here everyone.
KnockJohn
Sorry - our comments crossed. You and lots of others have been pointing out this not-especially-relevant fact. I am genuinely beginning to wonder why.
Let's stick to business.
BBD
"Spinning out the review process to delay publication by O’Donnell et al."
This, in my opinion, is the most pertinent point. The longer a paper remains unchallenged the greater and wider authority it gains. The Nature cover got a great deal of attention from the MSM which might never have happened had an early challenge to Steig been forthcoming.
billyquiz
Indeed. The longer a paper is unchallenged, the greater its influence. Especially if the challenger is carefully sabotaged by a reviewer so that he can attack it on publication. That is, if Plan A - kill it off during review - didn't work.
Not wishing to go too far OT but some of the comments above raise in my mind the question of who were the original reviewers of Steig (09) for Nature. Was there a statistician amongst them? Clearly Nature was getting a pretty clean bill of health to front page this paper, so it sort of makes you wonder.
I once suggested that the Team weren't much good at strategic thinking. I still hold to that, but there is evidence of improved progress: Steig shows promise, doesn't he?
I think we're getting side-tracked b focussing on the question if an author should review anonymously a paper critical of a paper he wrote.
The points are that reviewer A (Steig et al) used his position to block the publication of RyanO's paper, by the sheer amount of critique and counter-critique, in the hope that the delays would make them or the editor give up, and more importantly the further open critique of this paper, in spite of the changes he proposed, implying they do this or the paper wouldn't get published.
Now he's come out (see his reply at RC) with the statement that Ryan et al was a 'lousy' paper anyway, with the further obfuscation of going for the diminished 'warming', rather than for the statistical methods showing that this warming was originally only due to his incompetent statistical methods,
So let's not let them get away with yet another pea-shuffling exercise but hold their feet to the fire:
* abuse of peer review by trying to block a paper;
* abuse of peer review by using this information to criticise said paper;
* unwillingness to acknowledge that this was about statistical methods, not about warming.
@BBD, Billy Quiz - with respect and in my opinion, these are symptoms of the incident, not the proximate cause.
That was down to Reviewer A being in the position he was. Once he got into that position, the rest would have been a predictable possibility (or probability, depending on your level of cynicism).