Bishop Hill Moriarty on peer review
Apr 22, 2013 Journals The University of Nottingham's Phil Moriarty has written an excellent piece in Times Higher Education looking at post-publication peer review, whether through official channels or via blogs.
Responding last year to criticism of their field in the wake of the serial fraud committed by Diederik Stapel, three social psychologists - Wolfgang Stroebe, Tom Postmes and Russell Spears - published a paper in Perspectives on Psychological Science, titled “Scientific Misconduct and the Myth of Self-Correction in Science”. This provided compelling evidence that, across the disciplines, peer review fails to root out fraud. This is worrisome enough. Yet even basic errors in the literature can now be extremely difficult to correct on any reasonable timescale.
Meanwhile, Jonathan Jones points us to this post - at once appalling and hilarious - about just how difficult publishing comments has become.
Reader Comments (32)
J. Jones is doing science a disservice by not naming the journal and the authors.
I remember reading that article by Rick Trebino a year or so ago. If it wasn't so serious a problem it would be highly amusing. I must admit that I must have led a very sheltered life when I read it, as a few years ago I used to think that academic scientific research was carried out by people who did science with the aim of finding out to the truth and who were of the highest integrity. Climate "science" was a rude awakening.
Ed - The post Jonathan points to is not by himself - it is a depressing 'classic', also mentioned by Phil Moriarty, and written by Rick Trebino. I suspect that someone in Trebino's field would be able to guess the names of the Journals involved, the authors, etc.
Probably this www.swampoptics.com/PDFs/Comment_for_web_site.pdf
In fact you can find the - finally published - comment here. HAS links to a longer version of the comment (presumably one of the many drafts mentioned by Trebino) which has the further advantage of not being paywalled. More discussion of this Trebino Comment saga at...ClimateAudit.
Now about those 2 fundamentally flawed psychology papers...... (plus ethical failings and hostile conflicted authors)
'Moon' and 'Fury' - Lewandowsky et al...
must chase my concerns up with, journals, UWA, etc
Philip Bratby " Climate "science" was a rude awakening."
It was a rude awakening to many real scientists! Now I view climate science as I do astrology, tea leaves reading, etc.
"It was a rude awakening to many real scientists! Now I view climate science as I do astrology, tea leaves reading, etc."
ConfusedPhoton, with a true aim - shoots the arrow into the bulls-eye.
@HAS
I think to published one is this:
http://frog.gatech.edu/Pubs/Xu-Comment-OptLett-2009.pdf
The one you referenced was the original submission that had to be reduced to one page.
Unfortunately, I now regard any branch of science outside physics/applied maths/engineering as deeply suspect.
I think this will be climate science's lasting legacy to the world.
I see the good Dr. has come around since Holmes threw him over the falls...
Let's not be beastly to Climate Alchemists. Just rename the subject Applied Atmospheric Politics and make them into honest people..
Martin A.
I think you are saying climate science's lasting legacy to the world is that this AGW controversy has revealed all science as suspect (...why stop at physics as after Trebino's story?).
This is an interesting turnaround that has over the years become more persuasive to me. Previously, I saw the politicization of AGW ruining the reputation of science and encouraging/rewarding/spreading its corruption across many fields of science and all its institutions. Now I am seeing the AGW bubble more as a exemplar, or a coalescence, or a exacerbation, of corrupt practices entrench in the science culture across many fields. Perhaps it was a disaster waiting to happen. Science culture (and the reward systems etc) already contained the seeds of its own destruction. It would only be a matter of time before it found the the right cause, the right vector. And unity in an enthusiasm to save the world from this environmental catastrophy...well this was just the vehicle it required.
BernieL:
Would any current scientific practitioner comment on the extent of this problem in his or her own field?
@jferguson: on that line of thought, I have so far come across 13 errors in Climate Alchemy. I could be wrong but the evidence increasingly shows real data are at considerable variance to the GCM-predicted outcome, e.g.: http://clivebest.com/blog/?p=4871 shows Total precipitable Water is falling significantly.
jferguson: "Would any current scientific practitioner comment on the extent of this problem in his or her own field?."
In Chemistry it is virtually impossible to publish significant work without considering the possibility that someone, somewhere, will try to reproduce it. The authors should certainly repeat experiments themselves, and if a paper attracts a lot of attention then it is a certainty that others will try likewise, especially if patentable.
This is one of the self correcting mechanisms that is not always present in other disciplines. Of course it is not always the fault of the scientist if experiments take many decades, huge quantities of money, no proper control experiments, and relatively few practitioners make the field an incestuous one.
No discipline can be perfect, but Chemists learn to accept failure quickly or stop being Chemists. There is also more chance of finding an anonymous, genuinely disinterested reviewer in a larger discipline.
jferguson: "Would any current scientific practitioner comment on the extent of this problem in his or her own field?."
In Chemistry it is virtually impossible to publish significant work without considering the possibility that someone, somewhere, will try to reproduce it. The authors should certainly repeat experiments themselves, and if a paper attracts a lot of attention then it is a certainty that others will try likewise, especially if patentable.
This is one of the self correcting mechanisms that is not always present in other disciplines. Of course it is not always the fault of the scientist if experiments take many decades, huge quantities of money, no proper control experiments, and relatively few practitioners make the field an incestuous one.
No discipline can be perfect, but Chemists learn to accept failure quickly or stop being Chemists. There is also more chance of finding an anonymous, genuinely disinterested reviewer in a larger discipline.
jferguson, my view (as the author of > 100 papers - and two Comments, which had a less rocky road than Trebino's) is that scientists are by and large honest. That means: most researchers set out to be mostly honest most of the time. It would be impossible to do any research if you could not trust most of what gets into the literature. And there are also mechanisms to remove incorrect findings.
That being said, like any activity involving humans, scientific research clearly does suffer from a range of poor and corrupt practices. The idealistic view whereby peer review and replicability remove all problems from published science is laughable. Some of the problems arise due to dishonesty - people out to make money out of a discovery or to make a name for themselves sometimes cut corners. More of the problems don't arise from dishonesty, though, but out of people fooling themselves - and others - by not being self-critical enough. And the structures of how science is organized don't make it easy enough to challenge errors - as discussed in the post by Moriarty.
@09:44 Jeremy Harvey.
Thanks for the info Jeremy. I'll go and sit on the naughty step for a while.
Is this another misunderstanding of peer review, the old fallacy that if a paper's been peer reviewed it must be good? In most disciplines, peer review is reasonably light touch, the reviewers are not responsible for the author's work and make no such claim. They address the question, "is this paper (subject to our comments) suitable for publication in this journal?" The 'yes' they might give does not reflect badly on them when it turns out, for example, whatever was allegedly demonstrated or proved in the paper cannot be replicated, and its replication that is the acid test. The work is the author's, and the responsibility for it is his, not the reviewers, who do not themselves re-do his work nor analyze his efforts in great detail. That is not what they are there for. Claims that a work having 'passed' peer-review mean that therefore it must be good/OK/correct tend, funnily enough, to come from the climate science brigade, who seem to understand science as little as they do climate.
Apr 22, 2013 at 10:59 AM BernieL
Bernie, You have pointed out that what I wrote has another interpretation, more positive than I had in mind.
In one way, it seems a bad thing that trust has been destroyed, to the extent that if I see (for example) a news item saying that a scientific paper has shown a relation between wine consumption and a type of cancer, I now think "I'm not at all sure I believe that", even though the research was, in all probability, conducted impeccably.
The destruction of trust will be a harmful legacy of climate science.
But if, as you suggest, it results in increased scrutiny of science in other areas where fabrication of evidence, misinterpretation of results or acceptance of statistically dubious results as truth is rife, then some good will have come from the sorry story.
-------------------------------------------------------------------------------------------------------
(...why stop at physics as after Trebino's story?).
I think with physics generally, things will be corrected as others attempt to reproduce or build on published results.)
Yes, the peer-review process is fundamentally broken. That has been my experience as an author, as a reviewer, and as an editor.
My own comment which I found impossible to get published in Physical Review Letters can be found on the arXiv at http://uk.arxiv.org/abs/0704.2065 . In the end I published a slightly longer version as a paper in Physics Letters A, which can be found at http://uk.arxiv.org/abs/0807.1608 . I think it is now universally agreed that I was right.
Amusingly Ian Walmsley, who is thanked in both versions, worked with Trebino on the FROG technique; the world is smaller than one might think.
One might expect peer reviewers to moderate claims to what was demonstrated in the paper itself. Is this, too, an unrealistic expectation?
"a news item saying that a scientific paper has shown a relation between wine consumption and a type of cancer, I now think "I'm not at all sure I believe that", even though the research was, in all probability, conducted impeccably."
I'm pretty sure that most of those sciencey news items are nonsense. Personally, I think there are lots of similarities between all the research that leads to the 'diet consensus' (ie all the saturated fat is evil kind of stuff) and the research that leads to the 'climate consensus'
Rob Burton
For sure, take any media trumpetting of a scientific finding with a heavy dose of salt (if that isn't too bad for your blood pressure ;-) ). Chances are that the finding presented in the journal article are much more nuanced than the media-ised version (Marcott et al being a case in point, although it also showed a weakness in peer review, in that it allowed publication of a graph including results described in the text as 'non-robust' - I agree with J Fergusson that peer review should have moderated this point but my suspicion is that it had the opposite effect).
I'm not convinced that peer review is fundamentally broken, more that it is fundamentally mis-represented and has had a few high profile failures (i.e. failing to identify egregious errors). The idea that peer review proves a paper 'correct' is an absurdity that has been presented to the public at large for far too long.
The bigger issue appears to be defensiveness in journal publication - journal editors and publishers need to remember that much (probably the majority) of what they publish will ultimately be proven wrong (to a certain value of the term 'wrong'), but also that being wrong is not a crime in scientific research - you reach the best conclusion you can from the data you have available, but as more data is collected your conclusions are either strengthened or proven incorrect. Even an honest mistake is acceptable, as long as it is conceded and corrected readily. However, the rightness or wrongness of any particular paper can only be determined after publication and after the wider scientific community has had the opportunity to assess and attempt to replicate the findings presented.
Trebino's post would have been better if he had limited it to 1.00 pages.
The only "peer review" which would be credible would be for ALL the data, and any assumptions made, to be published for anyone to examine. Any government truly interested in the well-being of their citizens would insist that no 'science' could influence government policy which did not meet these requirements.
Unfortunately I don't believe our government meets this standard.
Climate scientists are giving stamp collecting a bad name.
The area that I am currently working in involves human subjects, being asked to undergo strenuous physical exercise. This is not without risks, hence we have to get our studies ethically approved (properly, unlike Lewandowsky, for example) and peer-review is nit-picking regarding methodology and analysis.
Much of the experimentally and statistically flawed rubbish that masquarades as climate "science" would be quickly spotted and stamped on.
The stonewalling methods described in Jonathan Jones' piece are almost identical to the rigmarole the BBC uses to fend-off criticism of its reporting bias. I know ... I spent nearly 3-and-a-half years trying to get straight answers... and failed!
@Blanchard
Even Wikipedia editors (the irony is clear) are able to see the merit of the following:
---
Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. He remarks,
There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.
Richard Horton, editor of the British medical journal The Lancet, has said that
The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.
---
http://en.wikipedia.org/wiki/Peer_review