Thursday
Jan272011
by 
Bishop Hill
Bishop Hill SciTech committee to investigate peer review
Jan 27, 2011 Climate: Parliament Journals This could be interesting:
The Committee has today launched an inquiry into peer review. The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.
The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:
- the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public;
- measures to strengthen peer review;
- the value and use of peer reviewed science on advancing and testing scientific knowledge;
- the value and use of peer reviewed science in informing public debate;
- the extent to which peer review varies between scientific disciplines and between countries across the world;
- the processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases;
- the impact of IT and greater use of online resources on the peer review process; and
- possible alternatives to peer review.
The Committee welcomes submissions from scientists whose material has been peer reviewed, those who commission peer reviews and those who carry out peer review.
The Committee invites all written submissions on any of these issues by Thursday 10 March 2011.
Reader Comments (19)
Interesting indeed. Let's hope the committee doesn't skim over these:
Does anybody know why this has happened?
Though I think its a great idea (and may prove very embarasssing to some disciplines hem, hem) it doesn't at first sight seem to be the thing that a Parliamentary committee would want to get involved with. I detect the hand of the esteemed Graham Stringer somewhere here..
No doubt Phil n Trev and Keef are already burning the midnight oil at CRU preparing their submission.
Or perhaps they'll just send a copy of their seminal work 'Climatology Pal-Review by Dummies', available from all good alarmist blogs.
I think that CRU and the UEA will keep their head down on this and I think that they would be very wise to do so.
I too detect the hand of "Para 89" in this one
Has Phil published his complaint yet about the amount of investigations that he has been involved in. It must be a terrible distraction from his work or are these questions from a different type of people?
This is becoming like a scene from "Revenge of the Mystery Mummy," where the jodhpur- &-pith- helmet-clad explorer sweeps his flashlight around the newly-opened burial chamber, first here, then there. LIttle is revealed other than scorpion droppings and spiders until suddenly the beam falls on the hideous visage of a dusty, rotting corpse.
Popcorn, anyone?
The MPs may want to speak to peer review definition expert, Phil "keep them out somehow — even if we have to redefine what the peer-review literature is" Jones.
And perhaps an examination of pal review and its relationship to peer review would also be fruitful.
The most important aspect of the failure of peer review as show by the Climategate emails is not explicitly mentioned: perhaps it could be stated as 'Whether peer review can act as a barrier to the publication of papers which run contrary to an accepted consensus'.
Hmmmm ....
What about submissions from those of us whose material has not been peer reviewed, and who neither "commission" nor "carry out" peer review - but who might have something to say about the committee's premise that peer review is used to "validate" scientific results?
Or am I taking this call for submissions too literally (and/or too skeptically)?!
@DavidC: Good point, peer review as a form of censorship, that would certainly be worth investigating.
For my own part I believe the scope of this inquiry is far too wide. In my limited experience peer review is simply a look through the paper to see if there are glaring errors, it might be different elsewhere, or in my case, in modern times, but I doubt it given the numbers of papers where glaring errors actual get through the process.
I doubt to, with the exception of the coterie of Team members and their followers, is it worse in climate science than anywhere else. We have seen quite clearly that these activists have tried to hijack the peer review process in climate science to suppress papers that might disrupt their conspiracatorial plans to make IPPC assessment reports as scary as possible, but there are hundreds of unreported papers being published elsewhere, which is why they try to block them from the IPCC review. For my part the peer review process:
1. Isn't rigorous and never has been, it's been dragged into the spotlight because the climate activists used it to try and prove anything written in the scientific journals must be true and that "the science was in;"
2. Scientists of all ilks when reviewing papers that didn't agree with their own theories have tried to block these papers through the peer review, not just climate scientists;
3. A committee that quite clearly, and correctly. drew the obvious conclusions that the inquiries they had been told had been set up to investigate the CRU had done no such thing and that parliament had been lied to but decided not to pursue the matter isn't capable of making judgements about the nuances of peer review.
Part of the problem is public perception of what peer review means and is intended to do. Climate scientists and activists have promoted the perception that peer review somehow blesses a paper a being gospel. This is not the case. Peer review is meant to determine whether a paper would be of general interest to the readership of the particular journal, contributes to the field, i.e., potentially advances knowledge, it's methods seem appropriate, it's conclusions are plausible, and it is free of common or obvious errors. None of this means that its conclusions are necessarily correct. The doesn't seem to be much of a problem for normal science, only post normal science where public policy or profit potential is involved. Climate science/policy and the vaccine/autism debacle are illustrative of both.
Geronimo said:
A dispassionate inquiry into what peer review is and isn't capable of and how it can be abused is surely a good thing. The investigation into peer review might well come to similar conclusions as yours once the process has been examined.
The peer review process isn't perfect but some in the over-political field of climate science have lauded peer review as all but infallible whilst making efforts to control what gets published. It lends an unwarranted certainty to climate science as the IPCC has distilled it.
I am wary of points 2 and 8 in the inquiry details. Peer review needs no strengthening. Journals rely on their reputations. Of late it would seem that reputation relates to the tone of a journal rather than how it conducts itself. From my very limited view of science journals as a bog standard member of the public, prompted by the global warming bandwagon, it looks like you can get almost anything published if you ask around enough. What appears to be contentious is high profile papers in high profile publications being protected from competing ideas *in the same journal*.
Looks like an even deeper black hole.
You put fraud allegations in, nothing comes out without a coat of whitewash.
I would like to see peer review in climatology done in the same way as medical peer review. Where there are two sets of reviewers who are completely independant and are not told who's work they are reviewing or where the money is coming from.
This would surely put an end to the suspicion of the hockey team peer review system, where it seems to be a meer back slapping exercise and jobs for the boys.
When you consider the problems that Spencer and Braswell had with there Climate Sensitivity Paper and the improvements made by odonnell, Jeff ID and McIntyre to the Steig paper - both delayed by one hostile reviewer - it would makes things fairer and more open.
http://www.drroyspencer.com/2010/08/our-jgr-paper-on-feedbacks-is-published/
http://climateaudit.org/2010/12/02/odonnell-et-al-2010-refutes-steig-et-al-2009/
The archiving and availability of all materials required to reproduce, verbatim, the claimed results of a statistical analysis seems a more important issue than any of the points they have chosen as their terms of reference.
It is always going to be difficult to stop overly friendly or antagonistic reviewers from distorting the process and perhaps inadvisable or even impossible to legislate against.
On the other hand, the UK could easily compel its own state-funded scientists to adopt rigorous standards of data/code archiving and disclosure, with the specific requirement of reproducibility, and a suitable disciplinary/retraction mechanism for authors and papers which fail to meet the required standards.
If the peer review process even could be changed, it should to be extended to include a review of all accompanying code & data, with specific emphasis on the ease with which the claimed results can be reproduced and the quality (comments, annotations, metadata etc) of those materials.
Don't know where this has come from (but Stringer is a good bet.)
But I don't think you have to be too paranoid to see a connection with several recent pieces (e.g. Nurse on the Beeb) extolling the wonders of Peer Review.
Call me an old curmudgeon but my money is firmly with the outcome that any comment not being "peer reviewed" shall be cast into the outer darkness forthwith.
And the "peers" being defined (for Climate "Science") as meaning CRU, the MET Office, the Royal Society with, perhaps a few Government "Scientific Advisors" thrown in to make the numbers up (literally).
Blessed is he who expects not. For he shall not be disappointed.
Some interesting thoughts on Peer review here:
http://noconsensus.wordpress.com/2010/11/23/how-a-scientific-integrity-act-could-shift-the-global-warming-debate/
@ David C | Jan 27, 2011 at 11:04 PM
'Whether peer review can act as a barrier to the publication of papers which run contrary to an accepted consensus'.
I agree with that point for some journals, although a paper can be published sooner or later in a different journal. After all, academic publishing is a lucrative business and it is in the interest of the journal to increase the number of publications.
What worries me more is the use of subjective and biased peer review to block research projects. Here the interests are the opposite of those in the journals: funding agencies are looking for any excuse to reject a project, therefore it is enough for biased reviewers to say that they don't like the project and the research proposal is scrapped. This is a very efficient way of 'manufacturing consent' and it happens very, very often. Contrary to journals, there is no right to appeal, or to proof the reviewer wrong. See for example the causes for rejection on Pielke Sr. projects
Below is an actual review that rejected a project for investigating very specific hydrometeorological mechanisms that affect snow cover and water resources. The reviewers were choosen among international experts. Somebody explain to me what is "the box". Needless to say that the project was rejected, and when complaining the PI was advised of the risk of unwelcome criticisms to the Pannel
Stress is theirs. If any mildly critical research or those that do not conform to the prescribed line of action and objectives are suppressed from scratch, it is no surprise that "the science is settled"
Jgc
In short how can is work be used to forward the higher political agenda behind the promotion of AGW. Or dam the data quality what use it for plugging AGW.
"the box" presumably refers to those constraints which have historically, and as a matter of good practice, applied to objective scientific enquiry. The reviewer is clearly unsettled by a proposal which is insufficiently strident in its commitment to supply readily cut-and-pastable headline quotes to waiting-in-the-wings eco-advocacy shills and carpetbaggers.
A wonderful, paragraph delineated, example of the juxtaposition of science and advocacy, nonetheless.