Thursday
Nov172011
by 
Bishop Hill
Bishop Hill Richard Smith: peer review still useless
Nov 17, 2011 Journals Richard Smith, the BMJ editor whose fulminations against pre-publication peer review have done so much to inform my own thinking on the subject, has returned to the fray:
Recently a paper that I wrote with several others was reviewed by another journal of global repute. Again there were three reviewers (the Holy Trinity), and I'm not very unkind when I paraphrase their reviews as: Reviewer A: "Please reference my work"; Reviewer B: "Pay more attention to my specialty"; and Reviewer C "The authors should have written the paper in the gnomic language that I use."
Last time I mentioned my views on peer review, Bob Ward said my views were biased and misleading.
Look out Dr Smith.
Reader Comments (33)
And what experience is informing Bob (failed scientist) Ward then? He didn't even get to publish a thesis.
I had a grant proposal rejected;
one really liked it,
one said it could not be done and
one said it had been done before.
I see the Guardian are reporting that the IPCC are expected to confirm link between climate change and extreme weather, which contradicts what Richard Black had reported earlier in the week.
http://www.guardian.co.uk/environment/2011/nov/17/ipcc-climate-change-extreme-weather?intcmp=122
The science is saying one thing, the politicians and environmentalists are determined to say the opposite.
It is a right old mess.
Don't you perhaps note a slight difference between your and Dr. Smith's views? He is interested in making more efficient progress in science, you are interested in finding reasons to ignore progress in science.
I would like to recommend his book "The trouble with Medical Journals". If I remember correctly Cambridge University Press found the first draft not academic, removing the stories would make it academic. I thank Richard Smith for choosing another publisher.
Frank
Please don't just make unsubstantiated statements like that. If you have a serious point to make, feel free. Otherwise please refrain from commenting.
Others
Please do not respond.
Harold
I cite the book in Hockey Stick Illusion. It's rather entertaining as academic books go.
If anyone is interested, link here.
http://www.amazon.co.uk/gp/product/1853156736/ref=as_li_ss_tl?ie=UTF8&tag=bishil-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=1853156736
Mac
Remember that the Summary for Policymakers is what is important. If the IPCC follows the scenario worked out for AR4 the SfP is published first and the main course follows later. The SfP included all sorts of political input from all the usual suspects (not to mention several unusual ones as well) and it was made quite clear that the main reports had by and large to follow SfP. Only "textual" alterations were to be made, if I recall correctly. The final drafting of the SfP did and no doubt will again exclude a fair number of the caveats that scientists will have included in their reports.
So it is more than likely that when AR5 is finally published it will include quite a few 'mights', 'maybes', 'very likelys' that were not included in the MfP (as was the case last time).
So there is no contradiction here. Black has got his hands on what the scientists are actually saying; the Grauniad (surprise, surprise) is plugging what the NGOs want the politicians to hear.
I am a recently retired physician. I once submitted a case report (2 cases) for publication. The point of it was to illustrate the difficulty that can be experienced in distinguishing between tuberculosis of the small bowel and Crohn's disease (an inflammatory disorder of the gut).
One reviewer said that it was obvious that these cases were TB all along and the other said it was obvious they were Crohn's disease all along. It was not published!
I thought this was a good example of slightly dodgy peer review...
Link
I've seen too much of the 'please reference my work' type referee. Given the widespread use of supplementary information (in real science at least) I don't see why the anonymous reviews and responses are not collected in the supplementary information for every paper. Seems simple enough - this information is already being marshaled by the publisher - just make it public.
munroad
I talk about TB at work and my colleagues laugh. They have no idea about TB... it is just amazing.
From a Amazon review of Horton's book:
BMJ published the Enstrom and Kabat paper, if I remember.
That is,..."From a review of Smith's book"
I can't understand the fuss about "peer review" and its supposed ability to validate research papers.
I have done my own share of peer-reviewing and had work of my own reviewed prior to publication in engineering research journals.
Reviews (in engineering learned society publications, at least) normally consist of a reviewer saying:
* The work is original, so far as I am aware. [Or, if not, references to papers covering the development]
* The work is sufficiently significant to be worth publishing [It is not trivial - something that anyone could worked out if they needed to]
* The work is likely to be of interest to readers of the journal.
* Adequate references are given to previous work on the subject.
* Suggestions for improvements in the presentation- where the explanations could be made clearer or where material could be left out.
* Any errors spotted by the reviewer.
Some reviewers, invariably anonymous in engineering journals, give insightful suggestions for further research or ask thoughtful questions that can result in great improvements in the paper.
Note that none of the above make any pretence that the work reported has been validated.
A different universe from "climate science".
Martin, I entirely agree as a long suffering peer reviewer in a different field, analytical chemistry your summary agrees with my own experience. As a reviewer I would not have dreamt of criticising the authors opinions unless they were actually contradicted by the data presented.
I had a quick look around for other discussions on peer review and found this paper
http://www.publishingresearch.net/documents/PRCsummary4Warefinal.pdf
linked from a thoughtful and lively article here
http://occamstypewriter.org/notranting/2011/01/26/more-peer-review-zzzzzz/
My own take is that if peer review is performed well then it is very useful. Whilst it can be quite easy to raise examples of bad peer review practices and climategate exposed some worrying examples of gate keeping and puffballs I still dont have any real view of how bad and widespread the situation actually is. Even with with climate science it is possible to find an example of constructive peer review prior to climategate.
As an illustration in 2008 Spencer and Braswell published a contoversial paper that led to this unpleasant "hit piece" by Raypierre at RC
http://www.realclimate.org/index.php/archives/2008/05/how-to-cook-a-graph-in-three-easy-lessons/
In a reply to this posted at RP Seniors site Roy Spencer made reference to peer reviews by Isaac Held and Piers Forster.
http://pielkeclimatesci.wordpress.com/2008/05/22/a-response-to-ray-pierrehumbert%E2%80%99s-real-climate-post-of-may-21-2008-by-roy-spencer/
Whilst it appears that Roy Spencer may have been a little out of order in naming the reviewers I have been interested in trying to establish their actual take on the papers. Isaac Held has recently broken his silence. For me it provides an example of good peer review where the reviewers have clearly been providing a constructive review that helped to improve the final paper.
http://www.gfdl.noaa.gov/blog/isaac-held/2011/10/07/18-noise-toa-fluxes-and-climate-sensitivity/
"This model is central to two papers by Spencer and Braswell (SB08, SB11). I provided a (signed) review for Journal of Climate of SB08, a fact that Spencer has publicized. My advice to simplify down to this one-box model, since it captures the essence of their argument, was followed by the authors. I was unclear on how best to estimate the ratio of the variances of N and S in this model — or, for that matter, whether this model was the appropriate setting for fitting to the data in question — so I didn’t address these issues in the review and recommended publication to let others work this out. I mention this since several people have asked me about my role in this paper."
The last paper I wrote, one reviewer provided a very good review complete with his work referenced at the appropriate point.
It got published.
@Shub. (O/T)
I've seen plenty of TB. Have a look at the TB Alert website. A third of the world's population infected, one tenth of them will develop TB. One new case infected every second.
I am just sending off the papers to apply to train as a speaker for the charity Water Aid.
The billions wasted on AGW alarmism could have had a major effect on TB, Malaria and provision of clean water/sanitation. Makes my blood boil!!
I agree entirely with Martin A, his description is what anyone involved in engineering would call 'normal'. The trick, in the different universe called climate science, has been to knowingly promote the false claim, a piece of sophistry to fool the masses which Lenin himself would applaud, that 'peer-reviewed = good, correct, must be right'. No it doesn't. It isn't meant to. It never did.
It was, however, an important claim to establish. Once established, and having also defined peer review as 'done by people we approve of in journals we approve of', it enabled the warmists to witheringly dismiss any rude-boy criticisms of their work, because having been peer-reviewed, that proves its right, right, so the rude-boy is either an ignoramus or a big oil stooge. And, most importantly, dishonestly inflating the value of the importance of peer review added significance to the equally dishonest claim, "all peer-reviewed science agrees ...." That people are prepared to lie to those who cannot judge the information put before them, for the sake of advancing a partisan position, tells you all you need to know about "climate scientists".
It was a serious point. That you are in complete denial of the difference between scientists and motivated reasoners like yourself is a shame, but hardly unsurprising. Feel free to banish me from your blog - echo chambers work better when they are empty.
Clivere
Read Richard Smith's book - peer review has been assessed scientifically and it doesn't work.
Frank
If it were a serious point, you would document where my alleged "motivated thinking" had led me to error. If you don't it's just verbiage.
Now, the subject of this thread is peer review. If you want to make serious points about anything else, please feel free to use the discussion forum.
Held's comment quoted by Clivere above seems to me actually to be a typical, traditional reviewer's position, that he doesn't necessarily understand all the finer points of the argument (because correlation between papers and reviewers is far from perfect) and anyway let it be published and see what happens. It is post publication that any real review happens (or not - the majority of papers are never cited) when others more expert in narrow fields than the reviewers see the flaws, or others try to replicate the work, if it is actually making such large claims that such an effort seems worthwhile. Peer review in most disciplines is a low threshold, and to make claims for a paper simply because it has got through peer review, is either deliberately dishonest or a simple misunderstanding of the processes of scholarly communication.
Speaking of Horton (as Shub had) ...
About the only redeemable part of the Muir Russell report was Appendix 5,Peer Review authored by The Lancet's Richard Horton. A few excerpts:
Horton also had a shorter version as an Op Ed in the Guardian (of all places!) on the day that Muir Russell report was released. He begins:
[More excerpts and source links at A catalyst for thorough reappraisal]
Andrew - I dont have the book so please provide the main reference(s) for "peer review has been assessed scientifically and it doesn't work"
Thanks
"and take peer review off its pedestal"
At the risk of being a bore, peer review is only on a pedestal for the mendacious and the uninformed.
Godlee et al is the main paper cited IIRC
http://jama.ama-assn.org/content/280/3/237.full.pdf
Andrew - thanks - I need to give it a more thorough read tomorrow and to see if I can identify more details of the paper and the 8 errors the authors actually introduced.
My initial reaction is it does not show that "peer review does not work" because 84% of the reviewers actually found at least one of the errors though only 10% found at least half the errors. Reviewer expertise and other factors including the nature of the errors will also matter.It does show a disparity about how thorough reviewers can be and does appear to demonstrate that different types of review dont produce widely different results.
I agree with Martin A, Arthur Dent & Bill that for most real scientific or engineering papers, you find that people are rational with their review comments. As a graduate student, I was often given papers that my professor was reviewing for my comments -- as part of my own training. Other students were also given the papers for their comments as well and we would then hold meetings to discuss them, their weaknesses and strengths. Just what went into the final review comments was for my professor to decide because he still wrote them by himself, but he generally held us to the set of guide lines outlined by Martin A.
I think Richard Smith's comments reflect sadly on the quality of those doing the reviews.
A common criticism from a referee is "Why is he tackling a problem that interests him rather than a problem that interests me?"
Bishop
This really, really bothers me because I have been involved with the scientific journal world and I know that at least in my little part of academia, peer review did work.
Perhaps it has to deal with the people involve. As surprising as this may be, most medical doctors are not scientists. They have gone to medical school and been trained as clinicians. Some do become very fine scientists, but just because they have an MD does not make them so. They have to learn that independent of their medical training. While medical doctors have some "scientific" training, that is they learn anatomy, the basics of neurology, pharmacology and a dozen other sciences, their real focus is in clinical work, one-on-one as patient and healer.
And, quite frankly, some medical doctors do become a little full of themselves. Perhaps it is because they save lives that they see themselves as god-like.
This not to say that scientists are immune from this hubris, because they are not, but it is much better controlled than appears in some medical facilities I have been in.
In my experience, peer review works reasonably well most of the time. Yes, some reviews say little more than "the author should cite my work" and some obviously haven't read the paper very well. And it's clear that peer review doesn't work well in a controversial area such as climate science.
But the weakness of Richard and Andrew's argument is that (at least here) there is no proposed alternative (remember the quote about democracy). My reading of the Godlee paper is not that it shows that peer review doesn't work, but that suggested alternatives such as making the authors anonymous or the reviewers nonymous do not improve things.
The comments here are interesting, in that they indicate that a high proportion of the regular commenters here are active scientists (Frank please note).
There was a recent Curry post on this.
Must go - just got an email "Subject: Review Report Overdue ..."
Paul Matthews - thanks for your comment. I no longer feel like a lone voice because I cannot agree with a statement that peer review doesnt work.
My own background is mainly in IT and I have experience of peer reviews from different perspectives and across a number of disciplines. I have seen very good peer review regimes but also some very indifferent ones
The academic world and climate science in particular does have some significant differences and clearly there are political pressures including many associated with the IPCC that may well have increased the ratio of poor reviews to good reviews.
In my opinion the IPCC review process clearly demonstrates a poor peer review regime where lip service is being paid to much of the peer review effort. However in this case it is probably not the fault of the reviewers but rather of the lead authors who appear to desire to provide their own simple sterile view for policy makers without representing all the issues.
To have a good peer review regime really requires all parties to be actively interested in making it work. In climate science I see a number of reasons why people may not always do the reviews proper justice and cannot see any initiative to resolve the pressures that are leading to this.