Bishop Hill Richard Smith on peer review
Dec 29, 2010 Journals I cite Richard Smith, the former editor of the British Medical Journal, a couple of times in The Hockey Stick Illusion. Smith was a pioneer of the formal study of peer review and his work has led him to believe that the technique is well past its sell-by date. He has recently published a very cogent summary of his views. Although Smith speaks naturally of the medical sciences, `Classical peer review: an empty gun' applies equally to other fields.
The article is full of good quotes. Take this for example:
Doug Altman, perhaps the leading expert on statistics in medical journals, sums it up thus: 'What should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal'
Reader Comments (35)
Ah, Peer Review of Peer Review -- about time.
I have been forced to collaborate in preparing medical papers of the sort referred to, and it is worse even than Smith paints it. Abstracts are almost routinely written before before the research is done so as to avoid "wasting" effort on research that isn't going to get published.
Also one sees tiny volumes of data massaged and coaxed and nurtured to support the pre-ordained conclusion (after all the abstract was written before the work was done). The end for me came when I was asked to fit a particular curve to just three points. Instead, I constructed a circle that fit them exactly. I was accused of sabotage, flippancy, and best of all, being unprofessional. Clearly my point was wasted on the individual concerned. Shortly after that I quit and got a real job.
'What should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions?'
For me, as bad as the rest are, the worst of all in the climate science area are the unjustified conclusions. Every time one looks into the methodology behind some broad sweeping announcement, the first impression is always "Really?!, You learned all that from so little?"
I used a Bland-Altman plot in my thesis.
Lovely find. I think on every point raised, I can think of examples in "Climatology"
"Doug Altman, perhaps the leading expert on statistics in medical journals, sums it up thus: 'What should we think about researchers who use the wrong techniques (either wilfully or in ignorance), use the right techniques wrongly, misinterpret their results, report their results selectively, cite the literature selectively, and draw unjustified conclusions? We should be appalled. Yet numerous studies of the medical literature have shown that all of the above phenomena are common. This is surely a scandal' "
I wonder if he's looked at climate papers?
One of the aspects of the virtuous corruption of virtual science is the effect of the communications revolution on peer review. The cost of communication has approached zero. The cost of international travel is much lower than at any time in history. At the same time, as knowledge has become more specialised, the size of any 'scientific community' of those with relevant expertise has become smaller. There is therefore a high probability that the 25 or so leading scholars will not only be known to each other, but will know each other personally. Chances are, they will have met each other at conferences and workshops and collaborated on research.
Add to this the fact that many of the journals involved do not follow double blind anonymous peer review (the authors are often known to reviewers - explicitly, not just through guess work), and you see the potential for 'pal review' and peer review as a gate-keeping function that we saw with Climategate.
Kuhn spoke of defenders of a dominant scientific paradigm, but Paul Feyerbend spoke of that defence involving all sorts of dirty tricks by scientists, from rhetorical devices to worse. Those are dsecriptions of how scientists behave. As consumers of science, we need to insist on a prescription based Popperian standards, including the ability to attempt falsifiction - which therefore requires openness, data disclosure, and the highest standards of proper peer review.
In Smith's field, let's not forget that there are protocols for the conduct of drug trials, etc that are designed to prevent even subconscious error creeping in to interpretation and analysis. In both chemicals regulation and pharmaceuticals we find things like Guidelines for Good Laboratory Practice developed in the OECD, and registered laboratories that are subject to regulatory auditing - with strong penalties for malfeasance. We hope scientists will behave well, but we assume some will not.
With climate science, the stakes are now just as high, but we let them manipulate data and run models unchecked by the sceptical, and engage in peer review. When found out (Climategate), we allow them to conduct tame reviews by people who are 'sound' and with appropriate terms of reference that are straight out of 'Yes, Minister.' We not only hope they will behave well, we pretend they all do - yet the warning signs have been there since James Hansen testified to Congress (Including Al Gore) in 1988 and the windows were deliberately left open overnight so that Washington's summer humidity would overwhelm the air conditioning to make for the appropriate televisual effect. (Does anyone else find it ironic that NASA, the space agency, eschews satellite data for a heavily 'adjusted' surface record?)
In short. peer review is part of the problem, but there are other parts. Does 'climate science' merit the name science?
More telling is the later quote: " While Drummond Rennie writes in what might be the greatest sentence ever published in a medical journal: 'There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.' " Could be speaking of 'the team' or CRU
My son, the statistician, has been telling me for years about the misapplications of statistics by the medical research community. He shares a similar view concerning the field of climatology.
Quote from the same author in a different paper
'The most famous piece of evidence on bias against authors comes from a study by DP Peters and SJ Ceci.6 They took 12 studies that came from prestigious institutions that had already been published in psychology journals. They retyped the papers, made minor changes to the titles, abstracts, and introductions but changed the authors' names and institutions. They invented institutions with names like the Tri-Valley Center for Human Potential. The papers were then resubmitted to the journals that had first published them. In only three cases did the journals realize that they had already published the paper, and eight of the remaining nine were rejected—not because of lack of originality but because of poor quality. Peters and Ceci concluded that this was evidence of bias against authors from less prestigious institutions.
This is known as the Mathew effect: `To those who have, shall be given; to those who have not shall be taken away even the little that they have'. I remember feeling the effect strongly when as a young editor I had to consider a paper submitted to the BMJ by Karl Popper.7 I was unimpressed and thought we should reject the paper. But we could not. The power of the name was too strong. So we published, and time has shown we were right to do so. The paper argued that we should pay much more attention to error in medicine, about 20 years before many papers appeared arguing the same.'
http://jrsm.rsmjournals.com/cgi/content/full/99/4/178
Using the opportunity of Dr Kellow's post, I would like to ask:
Has anyone in the audience at the Bish's blog had the opportunity to read the book Science and Public Policy: The Virtuous Corruption of Virtual Environmental Science?
Here's a very interesting article on the same theme:
http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/
O/T I can't believe a university vice-chancellor can be this stupid (my comments in the WUWT thread).
http://wattsupwiththat.com/2010/12/29/terence-kealey-what-does-climategate-say-about-science/
When people like this can seriously write this kind of thing the more I suspect all current science is bankrupt.
To be fair on medicine, it's come along way over the last century. I mean Doctors now cure more people than they kill.
Roy - "I was asked to fit a particular curve to just three points. This has never been a problem in climatology. They have state of the art infilling algorithms.
Peer reviewed of course.
Roy - "I was asked to fit a particular curve to just three points. This has never been a problem in climatology. They have state of the art infilling algorithms.
Peer reviewed of course.
timheyes:
Terence Kealey makes some good points, but he essentially commits the Naturalistic Fallacy by accepting what has been observed (scientists behaving badly) with what should be the behaviour of scientists.
Aside from the ethical dictates (honesty, fairness, etc) society does and should make demands on the conduct of science, which (while it might, as he suggests, deserve no public subsidy) is overwhelmingly paid for by the taxpayer. There is a prudential question about what standards we might apply to their conduct both as mechanisms of quality assurance and as a means of ensuring that all benefit from scientific knowledge.
Therefore, while Kuhn and Feyerabend describe the way in which scientists behaved in the past, society through its various institutions (eg the US Supreme Court in its Daubert ruling) have established standards as to how scientific evidence might be judged. The requirements are both ethical and prudential, and monopolies over use are granted and covered by laws such as those relating to patents. Patents, as a judge once recounted to me, were set at 15 years because two terms of apprenticeship were deemed to be an appropriate period for an exclusive monopoly. The example Professor Kealey gives of obstetric forceps being kept for exclusive use for more than a century has been deemed to be unacceptable because, as this case shows, the reward for innovation (in a privatised view of science) must be balanced against the public interest in preventing adverse neonatal outcomes (for mother and child).
Other kinds of knowledge are regarded substantially differently, even when the rapid dissemination of accurate information is a virtue: think markets and insider trading.
The practice of non-disclosure described does not apply under the Australian Research Council - and rightly so. Data must be archived and made available for reasonable use - even the documents and notes of interviews with which I deal. Those who generate data should have first use, but they do. I agree that it should be made publicly available at the point of publication, especially if it has been publicly funded. There is no justification for prolonged non-disclosure beyond that; the generators of the data will, after all, likely have other papers in press or in preparation by this time, and so any 'lie' will be halfway round the world while the sceptics are still putting their trousers (to paraphrase Churchill).
Even when science has enormous commercial importance, we regulate for quality and in the public interest. Pharmaceuticals and chemicals must have their efficacy and safety demonstrated under highly regulated conditions. Geological analyses on ore discoveries must meet appropriate standards of conduct and disclosure. Governments and stock exchanges insist upon it. Patent law provides commercial protection, but such protection also requires provision of evidence.
In short, climate science is overwhelmingly an enterprise that is paid for from the public purse. We have every right to demand openness and transparency and put in place institutional mechanisms that require good conduct, even if - indeed BECAUSE - Professor Kealey is correct in suggesting that they will behave badly. Some have misinterpreted Professor Kealey somewhat, and would do well to note the following statement near the end of his paper:
'More relevantly, it means that individual scientist’s pronouncements should be seen more as advertisements than as definitive.'
But I would point out that even advertisements are subject to the requirement not to make false claims, and advertisers should not be given a monopoly in any market. The alternative is the GUM store. He errs in not making the points I make here, and as a result, becomes an apologist for the indefensible. He should, rather, apply the standards of the USC in Daubert (or similar).
Nevertheless, as he believes that science requires no public subsidy, I take it we can expect that he will prohibit staff at his university from seeking public research funds.
Oops - screwed up with my new iphone
I was a Technical Editor on an International Medical Journal for four years (ironically under the aegis of the BMJ during Richard Smith’s tenure) and I fully concur with his views. My experience was that peer review was obstructive; time consuming, rarely objective and pedantically obsessed with minutiae that had little bearing on the paper itself.
I was appalled at how cynical authors were with the presentation of their data (often "salami slicing" one piece of research to get several papers with different Journals published) and how inept their implementation of statistical knowledge could be. When we could use qualified statisticians to review the papers, we did and they were invariably appalled at the quality of the work they encountered. We were quite unusual in our field in even bothering to use statisticians to peer review medical data at all which brings us to another of the glaring problems with peer review that Richard Smith didn't mention: the need to use the same reviewers over and over again.
Inevitably, the editor of a journal will need to stick to deadlines to get papers published and will veer towards a group of peer reviewers that he or she knows will do the job in a timely and professional manner. Although this may seem superficially valuable, it means the "gene pool" of opinion will contract to those reviewers that are used most often and whose views are hence recycled through every revision of every paper they encounter. However unconscious of bias the editor or reviewers may be, the results will be the same and the opinions of the industrious few will have an unjustifiable weight in all published papers and hence the “consensus view”. I hardly need to draw parallels with the AGW debacle.
Peer Review is not the only problem with scientific publications see this recent polemic
http://www.wca-environment.com/are-university-press-officers-bringing-science-into-disrepute-and-are-they-being-aided-and-abetted-by-their-academics/
Very interesting, but requires a lot of reflection. Do you think, Andrew, that you can keep it on the main page for a few days more so it doesn't get buried straight away in the archives the way most posts do?
Tangential I'll admit but the corruption of peer review is mirrored in the treatment of Tim Walker of the Telegraph by his fellow theatre critics in the London-based press. On appointment Walker refused to join the in-crowd's club - the Critics' Circle - and, worse, much worse, attacked the leftie luvvies running and writing for the taxpayer-funded British theatre. Walker's article is here:
http://www.telegraph.co.uk/culture/theatre/6907143/Vilified-ridiculed-ostracised-and-all-by-my-fellow-theatre-critics.html
I'm sure that peer review sceptics will recognise some uncanny similarities in the treatment of Walker to that accorded to climate science "outsiders".
Pirran
Your last paragraph raises a critical point with which i agree.
Although my experience is in a totally different and none technical field, my observations confirm that if you want anything done, give it to a busy person. As a result, those people are abused and the less productive people are given an easy ride.
Roy
I have been forced to collaborate in preparing medical papers of the sort referred to, and it is worse even than Smith paints it.
Yes, but it is nothing new. Many years ago after I completed my doctorate in physiological psychology, I interviewed a large US drug company with regard to working on psychotropic drugs, particularly with tranquilizers and such. It sound like the perfect job for me, given I had training in both biochemistry and pharmacology in addition to all the training in my major. That night I had dinner with the head of the research group who explained that their salary reviews were basically under the control of the marketing department. I went into computers instead. At least I can look at myself in the mirror each morning.
Publish or perish exists in all segments of research and too many "compromises" are made.
Methinks there is way too much reliance on peer review, reliance which supposes such review to be an expression of thoughtful scientific reaction to a paper.
The real question should be what happens when nonsense is published. My brother, the doctor, assures me that there is no public reaction to nonsense published in his field.
There can be little mystery as to why this might be true. One might even suppose that there is widespread abdication of professional responsibility for keeping the sophist noise level down.
Assuming the two things I've suggested above to be true, and accepting that the situation is evolving - hopefully - the activities at Air Vent, Climate Audit, and Lucia's may be where nonsense gets nailed. At least in the "climate science" arena.
Are there similar blogs auditing the publications in other disciplines?
Maybe we're bemoaning the ineffectiveness of a practice - peer review - which never did what we now seem to expect of it and at the same time are failing to appreciate the very positive things beginning to happen on the web.
We may come to see the important stuff on the web and not in the "refereed" journals. If that happens, there'll be little need for peer reviewers.
John in France:
"Do you think, Andrew, that you can keep it on the main page for a few days more so it doesn't get buried straight away in the archives the way most posts do?"
You might want to look into using an RSS feed reader to access the blog. That way you can directly access any of the 15 most recent posts with a single click. I use a Firefox sidebar add-on for this -- but there are a zillion other RSS readers out there. Modern editions of browsers can also interpret RSS feeds as dynamic bookmarks.
A Cdn. study of peer review quality (in medicine) found that it was much worse after about the reviewer's 5th year, and that no amount of coaching, encouragement, or rebuke could remedy this. An interesting aspect of the 20-30 yr. tenure of the CRU pal-reviewers! Their contributions are clearly now deep into negative territory.
Sorry, forgot the link for the above: http://www.sciencenews.org/index/generic/activity/view/id/47477/title/Peer_review_No_improvement_with_practice
http://www.sciencenews.org/index/generic/activity/view/id/47477/title/Peer_review_No_improvement_with_practice
" ... a major new study concludes reviewers don’t improve with experience. Actually, they get demonstrably worse. What best distinguishes reviewers is merely how quickly their performance falls, according to Michael Callaham."
i am not sure where you are going with your comments on peer-review.
The article cited is amusing, and points out problems with peer-review. However, it doesn't point out an alternative that is credible. Specifically, it doesn't point out an alternative that gets rid of the problems associated with peer-review, such as poorly implemented statistical analysis.
peer-review is good for what it does; but that doesn't mean peer-review implies perfection, or provides some other imprimatur. Criticising peer-review for what it is not, is not a sensible way to go.
Per
There is a section in the article on alternatives to prepub peer review, majoring on post-pub peer review. I think this is right.
Before 1950 everything got published. After 1950 there was too much science being done and peer review stepped in to help journals select the best stuff. Now we have the internet we can publish everything again.
That's my take on it anyway.
per;
I won't bother linking to the study, but a group submitted copies of existing published and praised research back to the journals with attributions and institutions changed. Only about 20% caught on, and only about 15% accepted the submissions; the rest rejected and condemned them as unworthy. Though they'd previously published them.
They also deliberately took published studies and re-submitted them elsewhere with deliberately introduced errors, small and glaring in various proportions. Again, about 20% were caught.
So, what was that about peer review "being good at what it does" again? The only thing it seems to achieve is filter articles by prestige of submitting author and institution.
without trying to be too cutting, it seems to me that "post-publication peer-review" means no quality mechanism at all at the publication stage. It is difficult for me to see how this is a step forward, but I note, for example, PLOS One, where all articles are published so long as they pass a "technical competence" peer-review.
The absence of peer-review at the pre-publication stage would make things really rather difficult in sorting the wheat from the chaff, in terms of scientific quality. There is the additional problem that all journals would have to change beyond recognition, and that scientists would also have to change enormously. Science in the UK is (I put this crudely) measured through your ability to publish in high quality journals (formally it is the quality of the paper, but the journal is frequently used as a metric of quality). Again to be clear, universities get money on the basis of this assessment of quality.
In short, i don't see post-publication peer-review solving the problems of pre-publication peer-review. References to the '50s are interesting, but the nature of science at that time, the small size of the scientific enterprise at that time, and the contemporary requirement for constant productivity (i.e. high impact papers) means that the comparison is difficult.
As for Brian H's study on peer-review, it is difficult to comment sensibly without looking at the details, and some of the details, like how you respond to referee comments, are clear variables. However, even if this study does show some sort of problem with peer review, what is the alternative ? I am unconvinced that the absence of a review process will improve the problem identified by this paper.
But my overview comment is that peer-review has known problems. It is more of a starting point for consideration of a paper, and establishing its scientific validity; it is not a guarantee of honesty, good science, or anything else beyond that it has been peer-reviewed. Quite often, peer-review is a useful process. It doesn't make sense to attack peer-review for being something it is not.
"It doesn't make sense to attack peer-review for being something it is not."
No, but it makes sense to attack it for not being something it claims.
What does it succeed at that it claims it does?
Here's the study: http://breast-cancer-research.com/content/12/S4/S13
Excerpt:
My %s from memory above were a bit off, but the points remain.
In a highly contentious and polarized field with big money at stake, like climatology, has "peer-reviewed" come to mean, de facto, "filtered, fudged, and corrupted"?
ok, so according to your quote, peer-review picked up a median of two (out of eight) errors. Why does it make sense to replace this system with e.g. an absence of peer review, which will pick up zero out of eight errors ?
in respect of the fifth point, i point out that there can be multiple interpretations of the same result.
Under any circumstances, you are NOT suggesting a better alternative. The absence of peer-review seems to me to have problems which may be potentially worse.
I think your "filtered, fudged, and corrupted" is not helpful, nor insightful.
per
per;
The evil that such p-review causes is not worth the candle. It suppresses disagreement with received opinion and theory, and is readily abused to play favourites with the editor's associates. Many instances of this have been documented. It also forms the basis of a widespread illusion that p-review assures validity, and indeed that ONLY p-reviewed material is worth any attention whatsoever.
The damage this causes is so severe that the minor irritation of having a few more errors make it into print than the huge % that already does is trivial. It would also put the onus for assessing results back where it belongs: on the readers and potential users.
Peer review has become a malignancy on the body and brain of science.
I hear a vast amount of colourful rhetoric and unsubstantiated assertions. So wild, that it isn't worth engaging.
But the point remains. While it is all very well taking a potshot at peer-review, it isn't a terribly principled thing to do if there isn't an alternative. Abolishing all peer-review is not accepted; there are things like Arxiv, and PLOS One, and vanity publishing, and they have not got the popularity of a paper in high impact peer-reviewed journals. There are good reasons for that. Simply asserting that doing away with peer-review is credible is a little bit like King Canute ordering the tide to stay out; you can do it...
If you are whinging about peer-review, because you don't like particular bits of peer-reviewed science, then it would look very much like an anti-science rant.
So I am not certain where you are going with the comments about peer-review.