Tuesday
Apr262011
by 
Bishop Hill
Bishop Hill SciTech hearings on peer review
Apr 26, 2011 Climate: Parliament The House of Commons Science and Technology Committee is going to take evidence for their peer review inquiry on 4th May. Those invited to speak are:
- Dr Nicola Gulley, Editorial Director, Institute of Physics Publishing Ltd
- Professor Ronald Laskey CBE FRS FMedSci, Vice-President, Academy of Medical Sciences
- Dr Robert Parker, Interim Chief Executive, Royal Society of Chemistry
- Professor John Pethica FRS, Physical Secretary and Vice-President, Royal Society.
Most of the members of the committee just seem to do what the whips tell them to, but it will nevertheless be interesting to see if they are going to simply go through the motions or if they have invited anyone who is a critic of the peer review process. Experience suggests, however, that the committee prefer not to hear anything from anyone who might rock the boat.
Reader Comments (15)
There is a saying among lawyers that you don't ask the witness a question where 1) you don't already know the answer, or 2) there's the possibility you won't like the answer. With legislatures made up mostly of shysters, what would you expect?
cheers,
gary
I wonder if this inquiry is why Oxford and UEA are currently testing the boundaries of FOI. During the Climategate whitewashes, it was suggested that peer review is the place for exclusive publication and FOI may bypass that. If the ICO does not rule on UEA's breach of it's agreement for Steve McIntyre's request, and the peer review inquiry gets a similar coat of whitewash, then climate scientists will be free to carry on BAU and exclude publishing 'intellectual property' that would allow their work to be validated.
A paper that passed peer review in Nature and got headlines claiming a 50% drop in phytoplankton has been now been shown to be fundamentally flawed. Yet don't expect this to be reported outside the blogs.
On Dot Earth, Revkin, after claiming that naysayers are breathlessly talking about this now, ignores that fact Willis Eschenbach pointed out the flaws last year using the simple smell test that there wasn't an observed corresponding 40% drop in the rest of the food chain.
At least he does quote this
http://dotearth.blogs.nytimes.com/2011/04/26/on-plankton-warming-and-whiplash/
Note that this is the first session of oral evidence and consists of the learned societies, expect further sessions to be announced later
I'm not sure what they're doing this for, no one is going to listen to them. They told the CRU to be more open and have been totally ignored. They were told Oxburgh would look into the science, i.e. the papers that were challenged and they were totally ignored, So what are they trying to gain from all this?
A confrontation between funders & spenders. The only question is who will blink first.
Who decides which witnesses to call?
Prima facie., these guys are most likely to be strong supporters of the status quo. They have arrived at their exalted positions in part via the existing mechanisms of print-based publishing and snail mail communications..With a fair degree of nepotism and deference to one's elders and betters thrown in. They are not likely to rock the boat.
I hope that at least Graham Stringer will give them some searching questions to consider about the recent pathetic performance of pal-review wrt climatology. And focus their attention on what (if any) mechanisms are in place to replicate, audit and verify published work.
But all this post-hoc examination by the legislators will do nothing to regain trust in the existing processes. Instant communications have overtaken them already. The old structures were predicated upon the premise that being a long way away meant that written communications took a long time to arrive. And that publication required typesetting and ink and paper...all of which were in short supply. Articles had to be limited to 500 words - not because brevity is good - but because there was a physical limit to the number of pages a volume could be constructed with.
All of these constraints are now removed. I can no more foretell the eventual exact shape of scientific publishing than the next man. But it certainly won't be based on a secretive cabal of 'right thinking reviewers' conspiring to conceal their own data and subvert their rivals.
Those days are gone. Technology has won. It is time to recognise this,
Embrace the technology, enrich the experience!
The RyanO et al paper was an interesting experience to watch develop from blog discussions to paper. Timeline was a little messed up when it entered the formal process, but when the reviews were revealed it provided a nice 'anatomy of a peer-reviewed paper'. The review comments added value to the experience and helped me understand where some of the controversies and uncertainties lie. Personally I'd like to see more review comments published if they add to understanding papers but understand the anonymity issues.
Another thing I liked about that paper was the turnkey code provided. I could do my own validation and play with the code & data to better understand what it was all about. It went off and fetched some data though, which got me thinking. A lot of climate papers are time series work, time moves on, data is updated, do papers conclusions still hold true? For a controversial paper like Jones and Wang's 1990 UHI one it could be possible to run off a 2010 version to see what's happened, if the code had been provided and the data existed. That should be possible to do for an online journal, but would need more effort to maintain data or links to data. Same seems true for some of the hockey sticks where they used old versions of data that have subsequently been corrected, adjusted or updated. Do the results still hold true? Online papers could be a lot more dynamic.
Something that may be simpler to implement could be citations. Jones & Wang has been heavily used in other papers. If it's ever retracted or updated, it may be nice to be able to flag papers dependent on the original with a warning that they may also need updating, or their conculsions may no longer be valid. A lot more seems possible in an online world though. Some downsides though, online or epubs would be harder to read Galileo style in a nice hot bath.
Technology doesn't provide quality assurance, which is roughly what people (including the govt) are after from policy-informing research. Efforts like this to consider and critique peer review are a good start for a bit of self-introspection by those who sit on the research->government channels.
Personally I think it's in the wrong direction (my written submission to the committee is number 72 here ), that is that concentrating on peer review as a means, rather than better qualified results as an ends, is misguided. But that may change; it's changed in some research disciplines that include medical sciences, some chemical sciences and some physics sciences, so the invited speakers might bring those changes to the notice of the committee. If these filter through to climate research all well and good, but that's only one of many research topics that feed government policy.
Professor John Pethica gave us this peach from The Royal Society's report "Climate change: A Summary of the Science".
There is no peer-reviewed science based on ice core data that supports this RS assertion that "changes in CO2 can lead to climate change and climate change can also alter the concentrations of CO2".
Indeed if this was the case then this planet would have experienced a runaway effect - postive feedback - instability that would have led to very high levels of CO2 in the atmosphere and very high surface temperatures a very, very long time ago.
The ice core records are clear - on this planet atmospheric CO2 is not a climate driver - it always lags changes in temperature.
There is scepticism in the IoP (I am a member) re AGW but their criticism of it is not vociferous. They are frightened of the politicians who will willingly withdraw funds from so many worthwhile research programmes.
I hope Dr Gulley will tell the Committee that AGW, carbon credit taxation and windmills are all scams.
I am not holding my breath on this.
1) This is not an investigation into climate change but an exploration of peer review.
2) The Committee Members read the written submissions, identify questions they wish to explore in more detail and then decide who they want to see in person to help with that exploration.
3) Witnesses at oral sessions are always grouped so that a particular topic can be explored. There is no surprise that the first session is given over to questioning the learned societies
Sorry folks, my earlier post is about topics that will not be considered by the Committee.
This came just to my inbox,
http://www.iiis2011.org/wmsci/Website/AboutConfer.asp?vc=27
"Empirical studies have shown that assessments made by independent reviewers of papers submitted to journals and abstracts submitted to conferences are no reproducible, i.e. agreement between reviewers is about what is expected by chance alone."
Hmmm, http://www.dilbert.com/strips/comic/2008-05-08/
"It is axiomatic in government that hornets' nests should be left unstirred, cans of worms should remain unopened, and cats should be left firmly in bags and not set among the pigeons. Ministers should also leave boats unrocked, nettles ungrasped, refrain from taking bulls by the horns, and resolutely turn their backs to the music."
- Sir Humphrey Appleby