Bishop Hill Improving peer review
Jul 15, 2013 Climate: MWP Discover Magazine considers a new proposal for eliminating some of the bias from the peer review process.
...scientific manuscripts should be submitted for peer review with the results and discussion omitted. The reviewers would judge the submission on the strength of the methods and the introduction alone. If they recommended publication, the authors would then send them the full paper.
The reviewers would then have a chance to change their mind and reject it, or ask for further experiments to be carried out, but the ‘bar’ for this to happen would be high.
This is quite a neat idea, potentially removing much of the bias in the peer review process. Other problems would, however, remain. For example, You can imagine a paper that began "we updated all the major tree ring temperature proxies in North America and northern Europe" being rejected out of hand - there are some questions that people just don't want answered.
Reader Comments (33)
It would require a half-way house to cleanse any information about the submitter so the reviewers have no clue regarding the source. It would not do to have a submission in the following way:
"Hey, guys Mike here. Give this a glance and shoot me your comments and lets get this rammed through asap. As always, a pleasure working with you. Kevin - good to see you last week at Cancun - next time bring some sun screen - OUCH! :)"
I've made up the names to protect the innocent.
Well, no. The most important part of a study is indeed the methodology. However, removing all else from a manuscript the paper does absolutely nothing to resolve the core problem, that is, the peer-review system itself. What we need is accountability and that requires a transparent procedure. Eliminate the false "blinds", I say.
"Discover Magazine considers a new proposal..."
Not exactly a NEW proposal - in the text it says:
...It’s a clever notion (and, as Smulders points out, not a new one; it dates to the 1970s, but has never taken off.)....
There's somehow been a misconception about "peer review" and that is somehow means that the paper is correct because it has been "reviewed". Peer reviewing is not the same as an engineering review, in that the content of the paper does not need to be correct. This may sound strange to anyone who hasn't published but the whole idea of peer review and publishing is that you put forward your work and as long as it's not glaring bad it gets added to the body of scientific information.
Whether it is wrong or right is taken on by people repeating the work or auditing it (a bit like what Steve Mc does). It's only in "climate science" that people have used peer review as being a measure of the paper being correct. And actually all this demonstrates is that people who do, don't understand the progress of science or have become advocates for a certain way of thinking and are just tallying up a "score" of papers that endorse their opinion.
Another thing: it is better to have papers out there that are dead wrong. Sometimes they get so many references because they demonstrate what NOT to do. The recent Marcott paper should stay as a shining example of good intentions but bad methodology and conclusions. You can then tell the calibre of people who reference it blindly.
Micky H Corbett,
Yes, it was just a way by which journals ensured they were publishing something new, interesting and not obviously tripe. Lots of peer reviewed papers were found not to be replicable or the experimental methods were shown to be invalid.
I think this idea of peer review as a stamp of undisputable correctness came from the arts. It feeds the idea that science is done by consensus of the wise and the good.
Agree with Micky H Corbett - In fact I would go one stage further in that because the peer review process became demonstrably "Pal Review" in Climatescience, that is why many started to question the "science"
How I cringed at the Alarmists twin mantra of "Peer Review = Consensus" which lead to the unbelievable "The Scientists have spoken!".
There are enough people in the hard scientific disciplines now looking at the soft science that is Climatescience for us to be reasonably confident that the truth will emerge.
No way, because the hidden results may no correspond to the described method. Far too much work for the reviewer, first he has to mentally co-invent the method along with the authors, then envision how the results should go, then, when he's already "reviewed" it, evaluate the actual results and fit them to his own preconceptions. Sorry, it's hard enough to get reviewers.
A good peer review would be independent experiments supporting a paper. Else. a warmomg, in font several sizes larger than the headline. stating "I'm Guessing. Maybe even SWAG."
I suggest that papers should only be eligible for peer review, once the data, codes etc are availvable, to ensure that interested parties have sufficient information to replicate the paper.
This procedure would ensure that the authors paid more attention to their work.
A bit like a mining prospectus, no?
Доверяй, но проверяй, comerade academician.
Some kind of review is necessary for any publication, otherwise complete rubbish from pranksters and mono-maniacs will swamp any useful stuff.
Most publications - magazines and newspapers, for instance, have their own reviewers on staff who know about their own specialist interest. Try getting a claim that you surfed a 200 ft high wave off Cornwall into a surfing magazine, for instance, and the editor will soon start asking questions. You won't get very far claiming that you have 10 1856 British Guiana 1 cent Magenta stamps in your collection to a philately magazine. It ought to be the same with ALL technical subjects.
The problem is that most of the journals we are concerned about are not run by people who love their subject - they are run as money-making concerns. Staff on ANY specialist journal should easily be able to identify suspect papers. Occasionally they might need to refer a piece to an external specialist - but they ought to know who the best specialists are for each issue, and be able to take responsibility for publishing controversial papers - that's why they are running that journal and maintaining it's reputation.
Newspapers are quite happy to work that way - they check out news and publish - and take the reputation hit if they are found to be publishing propaganda as fact. What is so special about Science journals which lets them charge huge cover prices, and then absolves them from responsibility for anything they publish?
Jul 15, 2013 at 10:31 AM Micky H Corbett
Agreed 100%. In past existences I have published plenty of papers which were refereed prior to publication and I have reviewed lots of papers for journals.
I had never heard the term "peer reviewed" until I started to clue myself up on climate science some years back. It seems to have originated with the IPCC boasting that they only refer to 'peer-reviewed' publications. The climate science community seems to have adopted it in the sense of:
"peer reviewed" = "validated and certified correct".
As someone on BH suggested recently, it might well become a pejorative term, in view of its association with "climate science". (The latter does not really, in my view, justify the term "science" in view of its reliance on the output of unvalidated models as "evidence" and its general avoidance of the scientific method.)
I'm 100% with Micky H Corbett here.
And I know of what I speak.
As others note above, a significant problem is how peer review seems to viewed by those less familiar with it. It's main value is a filter which helps save other scientists time, by not having to read so much that would really just be wasted effort. It's not a gold standard, it's a minimum standard. You still have to do your own thinking.
Irrespective, however well or badly peer review works, that won't stop exaggerated claims being made elsewhere.
How will that work? The reviewers want to check not just the method, but what conclusions (right or wrong) were arrived at from there, and the spin the authors put on it.
Peer review in climate science requires that the author (s) accept the 'Consensus'.
It's a bit like the Concordat in 17 th Century Rome insisting Galileo support the heliocentric view of the universe.
The implication is that "peer review" has some cachet, and we should be supporting and strengthening it, even though it says nothing about the validity of a paper.
I have no objection to peer review, but using it as a proxy for authenticity is a bridge too far. Let's maintain that distinction.
It is clear that the whole system needs tearing up and replacing with something not designed by Heath Robinson.
OK my name is Heath Robinson and these are my suggestions:
Several levels of review each one clearly defined and explained
Lowest: Peer Review. This paper has been "looked at" by a competent scientist to whom the conclusions seem reasonable and who found no obvious fault with the methodology.
Middle: Specific Review. Each aspect covered by this paper has been looked at by a scientist qualified in that field and who found the conclusions reasonable and found no obvious fault with the methodology. (All data and software used to be supplied)
Gold Standard: (payment of fees required) Fully analysed, Each aspect of this paper has been analysed by a competent scientist qualified in that field.
Data verified
Data selection verified
statistical/analytical methods verified
All results replicated
No fault can be found with the conclusions.
Relating to my Fully Analysed Review above, there would need to be some kind of accreditation system for reviewers and an appeal system both for authors and critics. a successful appeal showing a reviewer got it wrong would result in the reviewer losing his/her accreditation and the ability to earn fees.
+100 for everyone who has said similar.
Peer review is a workflow process in the academic publishing industry. Bit like stapling or loading in a van.
Anyone who thinks that publishing or peer-reviewing a claim makes something true has zero understanding of science.
Peer Review in most of science works just fine and does what it was originally intended to do; is this work novel, have the authors looked at other work in this area, have the provided sufficient detail of the methods for the work to be reproduced by others, are there any failures in logic in moving from results into discussion.
It "works" because the vast majority of published papers in science add very little value to the scientific canon, and in the majority of cases the work is of little relevance and rarely cited by others.
However, where leading edge scientific work is going to be immediately used for regulation or to support vast amounts of investment we need to have a more robust system that demands a higher level of proof. Climate scientists, NGOs and politicians still don't understand that data and conclusions need engineering standard review rather than conventional peer (pal) review if the public is going to tke the conclusions seriously.
Arthur Dent
The "engineering standard review" is actually a great analogy. If you consider the current spending on "climate mitigation" as a project spend to "solve an issue", then in any engineering review the first thing that is looked at are the requirements. And if they are valid.
Validity would come from supporting information and test data. In space projects this is called Pre Phase A data - the stuff that comes with a bid or at the start of a project when requirements are a bit more fluid.
The attempts to reduce CO2 warming would first of all need a good body of evidence that CO2 warms before money would be released. If you were to do this now, would you release the cash? I know I wouldn't with the current available data.
I agree with the comments above regarding efficacy of "peer review."
To apply an Americanism to this proposal, I think it is nuts. If one assumed the referees to be truly arms-length (couldn't identify the authors from the content), adopting this method would be to hold back information which might be required to determine the level of care needed for the review. Time would be wasted.
If I understand this genre of literature correctly, conclusions unsupported (unsupportable) by the research is common. Wouldn't one think that this form of error would be more readily detected by exposing the entire paper to review?
Yes, and Steve McIntyre's recent posts on the latest Briffa (ha) paper and the need to vastly improve the quality, quantity and transparency of tree ring data is, one must conclude, largely a work in the rhetorical style.
It is simply not in the interests of climate activists and their pseudoscience lackys to have any stage of the process be other than opaque, biased and politicised.
Many a career and pension entitlement depends on this state of affairs.
There is an incipient, general revolution in science, a NEW PARADIGM (yet very old; I call it "The Once and Future Paradigm"), that needs to come out, and peer-review -- any review by current deluded and dogmatically defensive "experts" -- is incompetent to handle it. Imagine back in Galileo's day, "peer review" was by ecclesiastic overseers; the situation is similar to that. We are dealing with an incompetent fundamental belief system, maintaining an incompetent consensus, across many fields of science, and climate science is just the tip of the iceberg, with its deluded assumption of a globe poised on the razor edge of "runaway global climate", either too hot OR too cold. No isolated debate, over any one supposed "cause" or "effect" (such as the "greenhouse effect" debate in climate science) can properly address the underlying general incompetence (that makes both alarmists and skeptics fellow-travellers in the "global-warming greenhouse effect" delusion). Adding epicycles to peer review is meaningless in that larger, proper context.
OK Harry I will just get my coat if that's OK?
You would need to have authors cloak themselves with noms de plume. Otherwise picture what would happen when Dick Lindzen submitted a paper to GRL.
No gimmick will work. The system needs to collapse first, which is where it is headed.
While I agree with Micky Corbett I think a simple solution to the issue is to have complete transparency in the process. The reviewers are named , their comments are published ( even if only in the online version of the paper) , the authors response to the reviewer is published and any editor comment is also published. This process is followed by a couple of journals that I know of -- I am not an author of published papers.
"Pay per view" ... reviewers should receive a stipend for undertaking a review, they should be named, and they should be accountable for shoddy outcomes. This will help replace some of the academic grant revenues that will evaporate when 'climate science (TM)' implodes.
Peer review = good/correct was a deliberate tactic by Schmidt et al, all part of the fooling of the public who could not know any better, to discredit any criticism of their work/policy agenda. In particular it was (as far as I remember) first wheeled out as a smack-down of the M&M paper published in E&E, the warmists wholly dishonestly asserting that E&E was not peer-reviewed, and using as evidence for that the then correct fact E&E was not in ISI, all journals in ISI are peer-reviewed, E&E not in ISI therefore not peer-reviewed. As 'Professor' Jones so succinctly put it, in a slightly different context "I'll re-define (what constitutes) the peer-reviewed literature if I have to", or words to that effect. In short, at every level, the warmist camp is shot through with intellectual dishonesty.
You say "The reviewers would judge the submission on the strength of the methods and the introduction alone. " I would continue this as follows:
Once approved, the work would be contracted out to a paid, independent third party; once completed, it WILL be published, no matter how it comes out. As far as the issue of how to pay for this:
Since no more work is being done than before, it shouldn't cost more than before. For example, maybe my graduate students will earn money by doing someone else's experiments, just as I will (in effect) pay someone else's graduate experiments to do mine. In fact, I would guess there will be less work done than before, since some "results" that were really just the result of absurd data mining or of experimental fraud would never have been proposed in the first place.
Sorry. One of the most important parts of peer review is deciding whether the conclusions drawn in a paper (especially those in the abstract) are justified by the data disclosed in a paper.
Peer review also needs to consider whether the appropriate caveats have been included and the results has been properly considered in light of the existing literature. Papers that produce unexpected or ground-breaking results are extremely attractive to prestigious journals.
If might be interesting if one reviewer were given only the introduction and methods. (The introduction and methods are what makes up much of a grant proposal, so that reviewer is essentially being asked whether the grant should be funded.)
In human clinical trials with new drugs, researchers are now required to prepare a site in a public depository for all the data they will collect, the methods they will use to analyze the data, and the projected date data will be deposited.
The whole "gentlements agreement" ethos of peer review is an anachronism. I understand how we got here, but If you want to clean up peer review:
1. Stipulate in the funding agreement for all government funded science that everything required to replicate the results be uploaded to a central on-line respository *prior* to press-release or publication.
2. The publishing instutution pays a repository a publication fee of £1,000 (to cover hosting and expenses) and a deposit of, say, £10,000
3. If the paper remains un-debunked after 12 months the publishing institutions deposit is returned. If it's debunked as garbage on WUWT within 24 hours the publishing institution loses their desposit (which is paid to the debunker) and the institution forfeits the right to publish to the repository or to receive further new government contracts for 6 months.
This would create a space in the market for business to be created employing scientists / statisticians / engineers etc as professional debunkers, profiting from the deposits from factually challenged authors.
This would also give the home institutions of factually challenged authors cause to look closely at the the scientific value of their staffs output.