Monday
Oct312011
by 
Bishop Hill
Bishop Hill Hulme on BEST and peer review
Oct 31, 2011 Climate: Surface Journals Mike Hulme has some perceptive comments about the BEST team's approach to peer review.
So what does this do to the conventional journal peer-review process? Those asked to review these manuscripts for JGR will now conduct their personal reviews in the full knowledge of the parallel public review which is on-going. And unless they shut-off all their communication platforms for the duration they will hear and see what others' judgements on the manuscripts are. Whether for better or worse it's difficult to see how this will not change the (conventional) peer-review process.
Reader Comments (37)
What kills BEST papers as they stand really is the additional complication of the PR Blitz - if I were a reviewer I would send everything back asking for Muller et al to specify every word very clearly, whenever Muller-the-scientist says something different from Muller-the-public-persona.
This can actually be a good thing...usually the "scientists" say silly things in the press release after the papers are published, when there's nobody to correct them.
If one were truly interested in the 'wisdom of the crowds' / 'public opinion' from the internet on the quality of the paper - I can't see how polluting the public arena on the results of the papers (whilst simultaneously handcuffing your reviewers) helps that cause.
Unethical behavior in my view, too studied for my taste.
"Let's see now," the reviewer asked him/herself, "which way is the wind blowing? Do I go on record as a fool for a political cause and the New World Order, or do I sacrifice myself on the alter of science? That is the question. Whether 'tis nobler to be a popular flash in the pan for a moment or two with Al Gore and the Good Old Boys of Global Climate Change and have my peers slash me to shreds as a fool for the rest of eternity, or be a real Scientist and tell the God's truth? Aye! That is the question.... Oh, hell, decisions, decisions! Alas, tomorrow is another day!" And he/she put the unreviewed paper back in his/her Hold Box and went on line to see what everyone else was saying about the Best Report.
Life's a real beach! There's only one way out.
New Satellite Data Contradicts Carbon Dioxide Climate Theory
Industrialized nations emit far less carbon dioxide than the Third World, according to latest evidence from Japan's Aerospace Exploration Agency (JAXA).
Global warming alarmism is turned on its head and the supposed role of carbon dioxide in climate change may be wrong, if the latest evidence from Japan's scientists is to be believed.
Japanese national broadcaster, NHK World, broke the astonishing story on their main Sunday evening news bulletin (October 30, 2011). Television viewers learned that the country's groundbreaking IBUKU satellite, launched in June 2009, appears to have scorched an indelible hole in conventional global warming theory.
http://www.suite101.com/news/new-satellite-data-contradicts-carbon-dioxide-climate-theory-a394975
Scientific Online Letters on the Atmosphere
http://www.jstage.jst.go.jp/article/sola/7/0/161/_pdf
http://www.jaxa.jp/projects/sat/gosat/index_e.html
You may wish to repost this in Unthreaded, OrkneyGal
Here's Maurizio Morabito being raked over the BEST coals on Skeptical Science:
http://www.skepticalscience.com/yes-its-still-us-and-its-still-bad.html
Looks like they make good points relating to his 3 questions to me.
I wanted to write something about 'Muller cornered' but I think our our host or Josh might make better use of it.. :-)
"our our" - please omit one. Either will do.
Orkneygal - Are they suggesting that there's a bit of a "nip" in the air.
There's been a lot of criticism, including from Judith Curry, of BEST's combined PR/review strategy. In terms of the content of the PR blitz, I completely agree that some of the "sceptics should now desist", "warming has not stopped" stuff is bad.
But I don't agree with the argument that the strategy for publication they have used is intrinsically wrong. We've discussed here many times that "peer review" is a kind of scientific shibboleth that does not deliver as much as its proponents claim it does. So we shouldn't get too uptight about people doing slightly non-standard things with respect to peer-review. This is all the more so that the "conventional" journal peer-review process is in fact remarkably variable, both in time (it has only worked the way it does now for 40 or 50 years), and in between journals and disciplines. Pre-publication on things like arXiv is well accepted in certain fields. In that context, the BEST FAQ response to the question "Why didn't Berkeley Earth wait for peer review?" seems fair enough to me. Everything that Hulme notes above would be true for any paper submitted to referees and pre-published on arXiv - the only difference being the greater amount of publicity for this work than for the average paper. But then to take another example, the OPERA paper on the possibly ultra-fast neutrinos is on arXiv, too, and that got quite a high profile also...
Jeremy Harvey sorry but BEST went out to whore the story to the press , not the other way around , they did this willing knowing how the press works , indeed the relied on this knowing that should these papers have real problems during peer review that story being 'old news ' is unlikely to be covered. So they got the 'message out ' in the way they wanted it regardless of the facts . The irony is that this happening in the past , to the determent of the science, was one reason for BEST in the first place.
The ultra-fast neutrinos is actual a good example of how it should be done , check then recheck they get others to check, then press release one you got others to check again while making the uncertainty in your results clear .
Sorry for the OT but just to underline how proud I am of having a SS post dedicated to myself. I thanked dana1981 already on my blog.
Strange how these high profile rows never make the BBC news though was delighted this was given a proper airing in Radio 4s review of the papers. Well done for once Auntie.
What Jeremy says. It's perfectly common to release papers publicly on a preprint archive at the same time as submitting them for peer review. It was inevitable that there would be huge fuss, whenever they released their results, and a huge amount of criticism. If the reviewers of the papers read what's been written in the blogosphere, I don't see that as a problem. There certainly are a number of problems with the papers and if Steve/Jeff/Anthony or anyone else point these out that can only be helpful.
But they should have been more carefully in their statements to the media, particularly with the remarks about 'no slow-down' which now seem to be contradictory, and the remarks about scepticism.
I wonder how irritated the editor of the journal which has accepted these papers, at least into its peer review process, is feeling. If he's very irritated, there's nothing to stop him changing his mind, refusing to publish on the grounds that his journal, like practically all others, generally expects submissions not to have been published elsewhere. Well since the papers are in the news, and on the internet, they've pretty much been published, haven't they?
Review be nonpareil
In this unparalleled gig.
Sunshine and suds, all.
============
@ Jeremy Harvey, I agree that the response sound reasonable. IMO , the problem is that the BEST approach was not preconceived. There is no mention of this in an earlier FAQ version.
http://web.archive.org/web/20110720061513/http://www.berkeleyearth.org/FAQ.php
The BEST team should have outlined their public response in a more transparent and apriori way.
Ross McKitrick on the CA "First thoughts on BEST" thread:
<BLOCKQUOTE> I agreed to serve as a referee back at the beginning of September and submitted my report almost a month ago. In so doing I accepted the journal’s request not to discuss the matter publicly while the paper is under review, and I intend to respect that commitment. It did not occur to me at the time that the BEST authors would fail so spectacularly to respect their corresponding obligations, or that the media, having shown zero interest in the many peer-reviewed and published papers on the topic, would so willingly join the tub-thumping on behalf of an unpublished, unreviewed PDF on someone’s website. </BLOCKQUOTE>
Pity the poor peer reviewer. He/she is a volunteer and is needed because of their particular expertise. How could a peer reviewer proceed in this case? What is he/she reviewing? Is it a paper BEST published on the internet? Is it that paper plus all the comments made by Muller? Yes, the peer reviewer can be legalistic about the matter, concentrate on the printed pages given him/her, and ignore the media and the gossip. But why would anyone who cares about the questions discussed in the article volunteer to be isolated from the ongoing discussion? Peer review cannot be practiced in this kind of weather.
The broad array of scientific journals, as they exist today and as they have for 100+ years, with their range of policies regarding publication processes and range of peer review processes, are not sufficient nor necessary for the scientific method and scientific processes to thrive. The journals are just an arbitrary social arrangement derived from historical factors.
If all scientific journals faded away, scientists would still seek funding (from private & public sources), scientists would still seek out the nature of reality and scientists would still find ways to share their thoughts and works. Science would still find media interested in the goings on within the scientific communities.
Journals are a convenience in some respect but not a necessity for science and scientists. Other forms of structure besides journals could easily be envisioned in our given culture's context.
John
How strange that a climatologist would show contempt for the peer review process.
Surely not an attempt to steamroller it through, before the IPCC deadline?
@Orkneygal Is it IBUKU or IBUKI?
John Whitman writes:
"Journals are a convenience in some respect but not a necessity for science and scientists. Other forms of structure besides journals could easily be envisioned in our given culture's context."
OK, but be aware that you are proposing a change. The only suggestion for a change that I see here or elsewhere is that journals will become blogs or be replaced by blogs. The proposal takes for granted that the value added by a journal and an editor is very little or will be replaced by something superior on blogs. My experience tells me that something really valuable will be lost if traditional journals and peer review disappear.
I agree, by the way, that the referees may be feeling annoyed about not being allowed to comment publicly. I'm not sure how this works out in fields where pre-publication is common. And though I can see that in a case like this one some communication with the press made sense, I do think that less spin on the results would have been good.
By the way, I forgot to mention that pre-release of results happens all the time: in conferences, people talk about unpublished work very often, and pretty often, one of the referees might be in the audience - this does not in practice destroy peer review as an institution. Nor is talking about unpublished work considered to be inappropriate - in fact, quite often it is a good way to get peoples opinions in a kind of extended peer review.
OK circulating pre-prints privately is common, and is one thing. But to stick the complete papers on a web site is essentially to publish them, is it not? So what then, from a journal's point of view, is the purpose in re-publishing them? A journal's aim is to publish original research, not recirculate research published elsewhere, surely? Most publishers can live with an earlier, less complete version, like a conference paper being in circulation, but most would jib at the thing itself being publicly available before 'publication'.
The idea, of massed peer revue, is good.
Throw the discussion open, "This is how we've done it, who can break it?" method.
That would have shown the issues with the proxies & data manipulation in the various Hockey Sticks up, very quickly, with the likes of McIntyre & Briggs looking at what's been done on what.
Now, would the peer (Or pal?) revuers, who were sent the papers by the journals, have had the cojonnes (sp?) to have picked up these and played them?
The way Muller's done it, is to trumpet his results to the press, almost as if he expects the peer (pal?) revue, to pass them through, largely unaltered.
That "Conventional" peer revue, is a faulty process, has been highlighted on many ocaissions, in many branches of science. Be that the MBH papers (& their spawn), the copy & paste from Wiki stunt that Wegmann performed, the cloning of Hwang and numerous instances in other areas of medicine.
Is it time (Or past time!) that a new approach, of massed revue was undertaken?
As Jeremy says, prepublication of manuscripts is very common in some branches of science. Indeed whenever I submit to Phys Rev A I do this by first putting the article on arxiv.org, and then the following day (after it has come out on arxiv) I "submit" it to the journal by the simple expedient of giving them the arxiv reference.
That's pretty much standard practice in experimental QIP. In theoretical QIP many people don't even bother submitting to a journal: the arxiv preprint often remains the only "published" version.
Probably does not change a thing. The people who know "peer review" is tenure-class fraud will shrug, the Believers will continue to believe, never seeing what the discussion is about.
@ Orkneygay
Thanks for the links to that fascinating report from Japan about new satellite data contradicting the carbon dioxide climate theory. There is a very interesting question in the first web page that you mentioned.
http://www.suite101.com/news/new-satellite-data-contradicts-carbon-dioxide-climate-theory-a394975
Towards the end of that page is a section headed:
Can Western Nations Still Proceed with Carbon Taxes?
Somehow I think the Japanese findings will get a very cool reception from Western governments, especially the British one. No doubt they will argue that controversial scientific claims deserve a very searching, sceptical response.
A conventional "peer review" is fine, an open peer review is also fine. But changing the chosen path as you hobble along is not. It is rude and it looks opportunistic.
Theo,
Hey, nice to see your comment. Thanks.
This thread came about from the actions of the BEST Project doing open MSM PR exposure of papers that are either under review at journals or are about to start the process at some journal. To me the only problem, other than BEST's inconsistent messages within their PR fumbling, I see is whether BEST violated the journals policy wrt paper going public during their pre-review or review processes. To me that is only an issue between BEST and the journal; with the journal taking appropriate action on any BEST project violation.
As far as now and the future, why couldn't classic journal peer review exist but also other processes offering alternate paths that voluntarily spring up from segments within the existing scientific community.
Broadening the whole scientific process discussion, one could ask whether there needs to be any guardians role/gatekeeping within the scientific process? I would think free association between all scientists without journals would finally produce alignment and agreement toward scientific progress. The argument could be made it might be more efficient than the current journal system. Great subject to think about.
John
John Whitman
it also means however that the selected panels of peer reviewers cannot join into the blog-a-along. So the papers are out there and the ho-ha is happening...if it turns out that these papers are crap, who will ever know. Whoever reported "global warming paper = pile of bullshit"? In this area, thje mass media are very important - David Cameron and George Osborne listen to no one else.
diogenes,
Thanks for your response.
I understand the existences of the peer review system and its general policies. But, I see no sufficiency or necessity in them.
Anyway, authors of the BEST Project papers should follow the policies (on pre-publication release of papers) of the journal(s) they submitted their papers to. If BEST Project violates the journal policies, then they need to respond to BEST about the consequences.
John
ORKNEYGAL posts links to some staggering new data from JAXA and the first responses are totally dismissive of her post. The BEST papers are about AGW and so are the JAXA papers.... offtopic??
Even if it was to have been off topic I for one would have welcomed it into the mainstream discussions on Bishophill.
I believe there are people here who love the very friendly (nay cosy) atmosphere in which anything and everything relating to AGW is discussed by intelligent and qualified people. However I believe that for some it is that discussion that motivates them and not science or truth. Were something to come along that would make these discusssions redundant these people would be gutted rather than be celebrating.
The JAXA data is (in the currently accepted debate) a true game changer that indeed may make these discussions redundant (at least in terms af any search for truth).
In reality (as we all agree on this blog) AGW was a straw man just waiting to be torched, it was a when rather than an if.
However all the JAXA papers achieve is to put one more piece into the jigsaw that will one day be a complete picture of how our climate works. Until that jigsaw is complete, each piece has absolutely no value in terms of predicting what the completed puzzle will look like. Instead of arguing that the straw man is indeed a straw man we should be arguing that anything and everything that any scientist currently says about how our climate works should be totally ignored.
Instead improve our economy and our education system, create the wealth that will make it possible for that jigsaw, one day to be completed.
Jeremy, you need to draw a clear distinction between the legitimate practice of circulating one's work for comment from colleagues, and going to the press to announce unpublished results. Nobody is criticising Muller et al. for releasing drafts and discussion papers (at least I'm not). It's common practice in economics and physics and other fields. But that is very different from issuing press releases, writing op-eds and giving detailed briefings to reporters. Defenders of the BEST strategy say the press would have reported on the drafts anyway, so they might as well go all the way with a media rollout. I disagree. Had the press picked up on the release of their discussion papers they could easily have said: These are just drafts, no further comment until they're in press. Most reporters I have dealt with would have respected that, and if some didn't it would be on their heads, not on the scientists. Any attempt to scoop the story would have been starved for oxygen if the authors had simply declined comment until after acceptance.
One of the lessons of the cold fusion fiasco was that circumventing journal peer review with premature press coverage not only deprives your colleagues of the opportunity to comment, but it can damage the reputation of the authors themselves, if the work is found faulty during the review process. Another lesson is that it is unfair to one's colleagues to expect them to engage in a detailed technical debate via soundbites in the media.
With Ross here. The pre publication is hardly an issue by itself.
If we follow the climate debate, it is quite clear that while journalists of all stripes are eager to write stories that favour the consensus, they would rarely do it until a scientist hands them a little bit of the meat and bones to build their story.
Ross, I think I did make that distinction (at least, I tried to). I said I thought their PR blitz was too much. To be honest, I think I can understand that they wanted to make some comment, but they should have been much more low-key, stressing that the papers have not yet been accepted, that the results need checking, etc. And I did also make the point that the situation must be annoying for referees - I'd seen your comment at CA.
Some people here and elsewhere have been suggesting that BEST's use of pre-publication broke some hallowed rule of scientific communication. I don't agree - though I do think they overhyped the PR with some indefensible conclusions.
Am very worried by the articles linked to by Orkneygal...will CAGWers next petition to cover all forests in asphalt, in order to save the planet?