Bishop Hill Praising post-publication peer review
Apr 7, 2011 Journals Richard Smith, former editor of the British Medical Journal and regular critic of the pre-publication peer review process, writes on the subject of post-publication peer review.
But more important than...formal types of peer review is the informal, the thousands of comments, decisions, and actions from the many that lead to a sorting of studies. I may hear a study presented or read a paper and be impressed. Others in the audience or other readers might also be impressed. We talk to friends about it. We email colleagues. We put it on listserves. Some of the recipients are impressed and start their own cascade. Others are less impressed and see problems. Perhaps a statistician attracted by the clamour reads the clinical article and sees important flaws that she shares with colleagues. Somebody might incorporate the study into a lecture, a review, or a grant application. And so a study might attract increasing attention and assume a prominent place, or it might fade as its receives more attention and more problems are noticed.
Many studies, in contrast, attract no attention—usually, but not always, rightly.
Reader Comments (15)
I think that much of the reason for peer review has been lost.
For the scientific process to work, all work has to be reviewed and tested on a fairly continual basesi, by anyone working in the vaguely te same area. This is how it has always been (always was ...).
The modern peer review for journals is an entirely different animal.
Modern journal staff probably know very little about the subject of their journal.
This means that there they have a problem: The volume of articles for publication and the strong possibility of publishing something which turns out to be complete crap,
To filter, they use "peer" reviewers.
That is really the only function of these reviewers, triage to select the more interesting papers, and a BS filter to keep garbage out of the journal.
This should not replace the real peer review, but has done so.
There is no forum for review other than the journals, and to get a critical paper published is not easy, because the criticism implicitly involves the journal and its reviewers - and they tend not to like this.
Quite honestly, the day of the journal is past. They just don't know it yet.
@ Philip Peake:
"Quite honestly, the day of the journal is past. They just don't know it yet."
I think you may be right. The "journal", as we know it today, is most likely a dying beast. The new journal is the comment provided in the blogosphere. This is the new peer review process. However is it often hard to distinguish the noise from the signal. I don't envision the paper journal disappearing entirely. It provides continuity from previous work to new work, without having to wade through the voluminous information available on the internet. I think the two forms can complement each other.
The theory, I guess, is better late than never. However, I for one do not think that scientific journals will disappear even though they may no longer be printed on paper. The issue is not what media the journal is published on, but the usefulness of the words themselves. Peer review at one time was an assurance that the material being published was worth one's time and effort to read.
Sadly that check and balance has been corrupted by Peer Pal Review and scientific journals with political agendas -- Nature is one such fallen publication.
However, I have faith that all bad things will pass and come to an end and new journals will come into being to replace those that have failed.
Perhaps I am being quixotic about it, but then I am Don Pablo de la Sierra for a reason.
I'm pretty sure the primary engineering journals aren't going anywhere, though whether they remain in print is a different question. I have about 300 lbs. of IEEE rags I wish were in electronic format. Of course, their peer review is significantly harsher and there are no shortage of experts to draw from for review.
Mark
Don Pablo,
I think pal review is a consequence of such a small field of "experts" to draw from for review. Inevitably they all know, and work with, each other. Something like 10,000,000 engineers in the US alone, 400k iin the IEEE as well. Big difference compared to the climate community.
mark
@ Philip Peake:
"Quite honestly, the day of the journal is past. They just don't know it yet."
It's the vehicle, the medium, that is in need of update. The Journal will endure. But the way it does business, the way that reviews are conducted, the vehicle by which it is "published" has all changed. Paper is out. Give it another 5-10 years to make all the changes and for the dinosaurs to die off. It's a done deal.
@ mark t
The number of members of the club is small. The field is multi-disciplined, but only those who think they know it all are tasked with review duties. As I see it, the primary discipline is statistical analysis. How many statisticians are members of the AGW club, though? How many statisticians have been asked to review the statistical methodology of the many papers?
The gross number of people in the field would be of no matter if the right people were asked to sit jury.
mark t
No, I disagree in the case of Nature a once prestigious journal, the one where great discoveries such as the structure of DNA were announced. Now it is run by a bunch of political hacks. However, one day they shall be shown the door. Post normal science is abnormal science, and too many of us know it for there not to be a backlash.
From the Ecclesiastical Uncle, an old retired bureaucrat in a field only remotely related to climate, with minimal qualifications and only half a mind.
Re Philip Peake, Apr 8, 2011, 12:03 AM and Gilbert K Arnold, Apr 8, 2011, 2:16 AM
Recently, from the ignoble basis of no prior experience I did some medical research. My docs thought the work well done and encouraged me to present it locally, which I did and where it was ignored.
So I considered the idea of publication and the internet was suggested. It was immediately obvious to me, however, that I wanted a peer review process in order to separate what I had done from tales of miraculous cures after hugging trees and taking snake oil.
This cannot be a new requirement. Some form of prior filtering will always be required.
From the Ecclesiastical Uncle, an old retired bureaucrat in a field only remotely related to climate, with minimal qualifications and only half a mind.
Re Philip Peake, Apr 8, 2011, 12:03 AM and Gilbert K Arnold, Apr 8, 2011, 2:16 AM
Recently, from the ignoble basis of no prior experience I did some medical research. My docs thought the work well done and encouraged me to present it locally, which I did and where it was ignored.
So I considered the idea of publication and the internet was suggested. It was immediately obvious to me, however, that I wanted a peer review process in order to separate what I had done from tales of miraculous cures after hugging trees and taking snake oil.
This cannot be a new requirement. Some form of prior filtering will always be required.
A little bit more on why I think the journal's day is done.
The original problem that they addressed was to provide a vehicle for the widespread dissemination of scientific papers. To make it financially viable the publication required a subscription. To keep subscribers, they had to ensure that content was worthwhile, which meant pre-publication review.
The world has moved on. There are better and essentially free alternatives for publishing. The limited number of pages in a journal and its low frequency of publication is a throttle on the dissemination of information, not the help that it once was.
I would hazard a guess that there is vastly more scientific information and research breakthroughs not published than there is published.
Peer review should mean widespread reading, discussion and feedback by the entire community of "peers", not a superficial quick look by a small cabal of "reviewers".
Regarding Richard Smith, former editor of the British Medical Journal and regular critic of the pre-publication peer review process, on the subject of post-publication peer review, I think he's got it.
For a glimpse of the tidalwave of the future of post-pub peer review look no further than The Bishop, Watts, and thousands of others. There isn't a lot of money in it, unless you have the backing of BP and other multi-nationals (and I really don't know anyone who has that), but it does reach out to the world and make a splash, doesn't it?
Don Pablo
I agree that this is true, but the fact that the field is so small all but guarantees pal review will result - whether Nature was run by hacks or people truly concerned about the science. Had the field been larger it may or may not have resorted to such a mess, though on that we can only speculate.
Mark
The vast majority of engineering breakthroughs are owned by private companies and, often, not even patented for fear of allowing others a way to implement them differently.
Mark
The days of the traditional journal may well be numbered (partly because they are simply so expensive). But the model provided by open peer commentary journals, pioneered by Harnad in the 1970s, will surely survive. Core target articles (subjected to peer review) are published alongside substantial critical commentaries (also refereed), with the core authors given the last word (i.e.right of reply). Behavioral and Brain Sciences is a good example: publication is highly competitive and the journal seems to be thriving. Crucially, you find high citation counts for core articles and critical comments. Altogether a healthy route for scientific publication. Where would climate science be if it had adopted this model?