Bishop Hill The problem of pal review
Aug 3, 2012 Journals The Committee on Publication Ethics has just recently reported on recent cases of misconduct it has dealt with, some of which are very interesting. This one in particular struck a familiar note, with an author of a paper trying to game the system by having his paper reviewed by sympathetic colleagues. In fact he even created sockpuppet reviewers.
On noticing a high volume of submissions from corresponding author A, editor X flagged up concerns with the preferred reviewers being suggested and their comments. Author A had in most cases suggested the same preferred reviewers for each submission, preferred reviewer accounts had non-attributable email addresses, comments were being returned very quickly (within 24 hours) and were often brief and positive, largely restricted to grammatical changes. All preferred reviewers favoured immediate acceptance or acceptance subject to minor revisions.
Author A was asked to provide further information on the preferred reviewers and admitted that these were either dummy accounts or associates of author A. The dummy accounts had email accounts accessible by author A and/or author A’s students or collaborators. Author A asked the preferred reviewers (or the people behind the accounts) to submit favourable reviews of the papers and turn them around quickly or author A submitted the reviews via the dummy account. Author A admitted employing this system for a number of papers, but not every paper, although we found similar patterns of peer review activity for these also. Author A states that the papers’ co-authors were not aware of this activity.
The verdict was damning -
The Forum agreed that there are many issues involved here, not least a serious form of misconduct which may even be criminal, as the author was impersonating the reviewers and committing fraud by using colleagues as false reviewers and, possibly impersonating other reviewers. In addition, as the author has admitted fraud, can the editor trust the validity of any of the papers?
Note, however, that COPE is not a court, merely providing advice to journals on how to proceed.
[I've altered the wording of the first para slightly, just to make it clear that I don't think that climate scientists have used sockpuppet reviewers]
Reader Comments (20)
The problem with pal review is inherent in the term 'pal' review, which is cronyism writ large, and that ain't 'review.'
It would be untenable to label all peer review as pal review, even reviews by those involved in climate, but this episode clearly points towards a more open system of review.
Ok slightly off topic but there is a web site that budding screenwriters use to get reviews of their scripts from other authors a bit like peer review, the sceenplay reviewers is unknown to the author prior to their reviews and suggestions being submitted. To get a screenplay out for review you have to prove your commitment to the process by submitting enough quality reviews to earn enough credits to publish.
I don't see why this approach cannot be adopted for peer review: http://labs.triggerstreet.com/labs/HowItWorks?ID=ReviewandCreditProcess
Pray tell me Sgr Galileo, as 97% of all priestly members of the Holy Catholic Church agree that the Sun revolves around the Earth, how can you argue against such a consensus? Furthermore, how many of your peers are willing, on pain of excommunication and eternal damnation, to agree that your writings on the contrary hypothesis be officially published?
To allow you to reflect further and to celebrate my election to the Jesuitical Committee on Frogs Causing Climate Disruption, I have ordered the thumb screws be tightened only half a turn today.
"I don't think that climate scientists have used sockpuppet reviewers"
Didn't one of the CRU "scientists" suggest altering the peer review process? Anybody remember which one?
Sockpuppet reviewers? Certainly not. They just got Editors fired.if they didn't choose real reviewers more carefully.
Despite a history of having published many papers (and of being a reviewer myself) I have never heard of an author suggesting the names of suitable reviewers. How amazing. Clearly I have led a sheltered life.
I'm surprised that the editor did not say something like "Thank you for your submission. Please note that our invariable policy is to select appropriate reviewers ourselves and strictly to maintain their anonymity".
How amazing.
My experience is that reviewer X MAY be suitable. That's all, editor to choose though.
@Martin A,
It happens, particularly in areas of research where sufficiently knowledgeable peers are much more likely to be known to the authors than the editorial team. I turned a review down recently because I was involved in setting the research project up!
Asking authors to suggest reviewers slightly suggests there's something wrong with the editor or the journal. Given that there is at least one journal for any micro-field imiganable, why does the editor not know of a few people qualified to comment on a particular paper? Or is it the case that a journal has a very broad scope so the editor can't possibly have connections in every sub-field covered by the journal? In which case shouldn't it consider narrowing its focus? The usual virtue of review is that it is anonymous: asking authors to suggest reviewers subverts that, invites corruption. On the other hand there is the argument that Dr x loathes Dr Y and will always trash his papers, given half a chance (eg as a reviewer). But in the micro-worlds of academia this kind of 'personality conflict' is usually well-known, and good editors take account of it. Overall, asking authors to suggest reviewers can be acceptable under very unusual circumstances. If a journal habitually does it, that should constitute a warning flag.
No one has suggest 'pal review ' is unique to climate science but it does seem to endemic to 'the Team ' who seem to regard only themselves as 'expert enough ' to review their work.
Perhaps what Mr Watts has done is the future of peer review...especially in fields where the established view is so rabidly aggressive against anyone that dares question their religious beliefs?
Mailman
This is always going to be issue in peer review. With the way the internet has opened everything up the masses we will inevitably move towards a more 'open source' system once the current model crumbles a bit more. Some people say that it will be too chaotic, but despite the potential problems the current open source software system works very, very well. It also works very well side by side with commercial closed source software, so perhaps there will be room for both systems to carry on together.
It's fascinating what has developed via the internet. My foray into climate science mirrors that of Claes Johnston [Sweden] and Doug Cotton [Australia]. We each concluded the 'consensus' belief about heat transfer at boundaries was wrong and this meant formalising physics Planck initiated but did not finish.
I very much appreciate the openness of Judith Curry and Roy Spencer who bravely debated their thinking. The reaction of supporters of the 'consensus' was to claim they had used the 'text book' answers but they had forgotten to look at the small print! There was active censorship in the MSM.
We might still be wrong but had the internet not been around, this and 4 other mistakes in this very important science would have remained unchallenged, not the scientific method I was taught.
I have never seen peer review as a gold standard myself. It can be a useful tool, especially in disciplines like Philosophy or History, but it has always seemed dubious to me in any area of hard science, good only for screening out the most obviously faulty papers. It has always suffered from a degree of pal review and much of it, even well made reviews, is of the, 'well, the methodology seems sounds, the results seem statistically valid, so yes, OK to publish, might be something in it' type.
The true gold standard in any scientific research is reproducibility, only reproducibility is truly valid, but not all scientists, and not only climate scientists at that, really welcome it if for no other reason than, as Phil Jones told Steve M, more or less, 'I've spent 25 years building this, why should I give you my data so you can poke holes in it?'
"I have never heard of an author suggesting the names of suitable reviewers": nor, for many years. had I. But it had become common in the decade or so before I retired.
Maybe the academic world needs to start again using the business model.
In order to be able publish a paper in a registered journal (meeting min standards), the author (or sponsor of the paper) should be required to employ an independent verifier from an accredited auditor (PWC, E&Y, KPMG .... etc etc) to test and vaildate the assertions and data.
These audit companies already carry out environmental audits with trained verifiers - it wouldn't take much for them to hire suitable academics to a professional, independent and trustworthy job.
Aug 3, 2012 at 8:42 PM | Chairman Al
You missed out Andersen's from your list....... But at least they suffered extremely badly when seen to not do their job properly.
Actually it is not uncommon for the editor to ask the author to suggest reviewers, particularly if the paper is outside the "mainstream" of papers published in the journal in question.
I have suggested reviewers on a few occasions, though I strongly dislike doing it and always indicates explicitly why I consider a particular reviewer to be suitable.
Asking for suggested reviewers by lazy editors is just one more side effect of having way too many papers and too many journals, and the nonsense culture of "publish or perish".
On the other side, the EGU journals approach is a good solution (IMHO). There the review process is completely public, even when reviewers can be anonymous, the public can contribute to the review and revies are archived and openly accessible.