Bishop Hill The press and scientific papers
Oct 21, 2011 Journals I've been enjoying the back and forth on the BEST thread about the way the publicity for the team's papers was handled, with some people concerned about the team going to the press before peer review had taken place.
Circulating drafts of a paper seems unobjectionable to me - this is surely an everyday occurrence in the academy. Going to the press before those drafts have been examined seems somewhat more questionable. That said, given my own views on peer review - namely that it's not worth a whole lot - then some interesting questions are raised.
If we have a world in which peer review is required, then is one of the roles of the process to tell the press that it's OK to talk about a paper? I would say that it is, in which case I think there is a problem with the publicity drive.
But how would things work in a world without peer review? - an open-source world in which papers might simply appear on scholars' websites for anyone interested to review and comment on? How do the press know that it is OK to discuss the paper, bearing in mind that what they write about the paper is likely to be taken as gospel by policymakers?
I imagine one possible measure might be the number of citations - the press would have to adopt a self-denying ordinance that would prevent them from discussing papers with fewer than five or ten citations. Then again, would it actually matter if they discussed papers that were shown to be nonsense days later? This happens, even with peer review in place. Would the press have to become more cautious about "extraordinary claims"? In my view that would not be a bad thing at all.
Reader Comments (19)
If you changed the model you would change everything not just one element related to media.
The reason MS was/is so scared of Open Source is not that the software was free as in cost. But that the model broke their feeder system where any number of middle men take their cut. Microsoft is built on over priced software where the consumer pays over the odds for everyone's cut along the way.
And due to the clever GPL licence you couldn't actually buy out the "owner" to stop this at source.
What would happen in an Open Scientific Information world?
Well it would driven by survival of the fittest. It would be evolution in action. Where your asset is not the paper you write, but the knowledge you use to write it.
Like the formation of a Universe, clusters of knowledge would be formed, gravitating as necessary. Breakaways (forks) could occur when at any point if new knowledge was not being accepted or recognised or was being politicised.
And just those who contribute to this universe? Well they would be judged purely on what they contribute and what they produce, rather than the letters they have after their name.
I could not see a true Open Source model, for scientific knowledge sharing, work. But fun to think of it none the less.
Good morning, Bish,
I don't understand the first clause in the last paragraph. Is there a word or punctuation mark missing? [Thanks, JiF- now dealt with. BH]
JiF
I think the main problem with the model outlined by Jiminy is that is is basically trial by popularity. In the world of software, utility drives popularity - if a piece of software is useful, it will become popular. If it's cheap or free, it will be more poipular still The open source nature of its creation only ensures consistency with current popular methodology/ideology in programming, not how popular it becomes. It's popular not because it's open source but because it's free (both in terms of cost, and in terms of commericla lock-in)
Trial by popularity in Scientific papers would (again) depend on utility - this time, how useful a paper is for obtaining the goals of the person rating it. For pro AGWers, the papers supporting it would become popular. For sceptics, the ones casting doubt on it would become popular. The two sides would spend all day throwing paper citations at each other. Sound familiar? This is how the blogosphere works right now, and we're getting nowhere with it.
I think there's a real danger in verging into the deeply UNscientific area of "my truth is just as good as your truth, I don't care where you studied" - it's a bit like market capitalism taken to the world of facts and knowledge. If you're cunning and stealthy enough, your 'truth' can come out on top, even though it may be an inferior 'truth' to the one held by most people.
I'm not arguing that peer-review as it is done at the moment is a Good Thing. I would go halfway to what you suggest - leave the writing of the papers to the professionals, but open up the review process to everyone. Before acceptance of a paper by a periodical, the authors must put it out for a period of public review. Make this count as 'publication' for the purposes of precedence (I know they worry about it) but make it unconfirmed until all objections have been dealt with. Each objection could be rated by popularity, so that obviously stupid ones could be discounted, leaving only the meaty objections to be dealt with by the author.
"peer review" is a workflow stage in the academic publishing industry. It's exists only in an effort to check that a paper is fairly novel and doesn't contain too many stupid mistakes.
It's part of the publishing industry in the same way that stirring ink and trimming paper are.
It's nothing more than that.
At some stage it would be good to find the official stance on peer review as it is confusing.
Recently I had a look at a Hanson paper, re BBD in Discussion, for interest.
MilankovicPaper
Just to follow up on comments about the paper I thought I would find where it had been published and see who had passed comment:
arXiv
Not previously knowing of arXiv further searching brought up this:
PaleoAndImbalance
The fact that you could cite a paper that hadn’t gone through a journals scrutiny made me wonder what the review process was for arXiv.
Endorsement
What are the responsibilities as an endorser?
Not being a part of, and I guess not understanding the scientific process , I am asking if this is normal to be able to cite a paper that hasn’t undergone formal review because so much emphasis has been expressed about papers that have undergone ‘proper scientific scrutiny’?
News stories are based on extraordinary claims, be they truths or lies. That is the nature of the beast that needs to be fed every day.
If you were really interested then you could try and assess how many papers from Atmospheric Chemistry and Physics Discussions (ACPD) get press coverage. ACPD publishes non-peer reviewed papers online that are then anonymously reviewed as well as accepting open review comments online. The subsequent revised paper is published in the proper journal, ACP, if it deals with the reviewer comments adequately. I can't imagine anyone would write a press release for an ACPD paper (the equivalent of the BEST move) but it would probably only take a bit of googling to find out.
restrictions on discussing science according to peer review? that's a new one. surprised it comes from a libertarian source. and it would do us skeptics the world of good, wouldn't it?
Just as a follow up, The Case for Young People and Nature by Hanson has 183,000,000 hits in Yahoo search.
Cursory look at a few turn up religious and socialist sites presenting the results to their readers as though from the mouth of God.
I'm not saying that there are any flaws in the paper as I am not qualifyed to critic it, but has anyone, is this post modern science?
From today's The Times;
"Climate scientists vindicated when disputed evidence of global warming is confirmed in a study funded by climate sceptics"
Better informed journalists might be a step in the right direction.
Why do scientists always seem to assume that announced findings are correct? We have a long track record of botched studies, especially in climate science. It's as if these idiots are incapable of learning.
I'm not at all troubled that there would be a PR release concurrent with these papers being turned loose in public. It doesn't offend me that the BEST folks would want their take on the significance made public at the same time as the papers simply because people who can understand the papers will not be the only ones wanting to comment. The press, in some cases, needs the training wheels a PR statement represents.
Far better to do it this way for everyone. Muller even entertains the thought that problems will be caught and may be fixed prior to formal publication.
Can dear Anthony really imagine that they would have floated these pre-publication without a PR?
or maybe, once again, i didn't understand any of this.
It is common in physics (and economics) to make papers publicly available at the time of submission; and it is common to draw the attention of peers to such pre-publications. This maximizes feedback.
It is not common to draw media attention to pre-pubs; and indeed this is often frowned upon as the paper has only been through internal quality control.
In this case, though, the blogosphere would have jumped on the BEST papers once posted on arXiv, the MSM would have followed. The authors decided to control the release of information to the public. I think this is justified in this case.
I think the release of the BEST papers is actually a pretty good "IQ Test" for scientists. Reading the quotes from other scientists who claim these studies prove far more than they even purport to address (assuming they even end up being correct) tells us all we need to know about the rational thought processes of these scientists. Every time one demonstrates the inability to focus on the limitations of the work, we are seeing another good reason not to trust their own work and the assessments they make about the state of the science.
If the police were seeking a hit man that witnesses described as tall and blonde, we would hope that the police chief would not be so stupid as to announce that the case was solved because a blonde man was taken into custody for questioning. Apparently, our society requires better analytical skills and rational thought from our police chiefs than we do of our climate scientists.
I really can't get to excited about the BEST study. It is a major league data mining and statistics effort that has given results that are in line with most other studies.
It shows that the land temperature record has warmed by about .9 oC over 50-60 years. Most of us would not dispute this. If "deniers" believe that there has not been an increase in the instrumental temeperature record, it is unlikely that they will swayed by this.
I think there are several caveats. First, the overall data quality is not particularly high. I fully acknowledge that staistics is the art of making judgements in the light of variable data, but this should be borne in mind.
Second, the confidence limits expand rapidly as one goes back in time. For example, can one say thatthe period 1960 - 2010 is comparable to 1850-1900? I very much doubt it because there simply isn't enough data to make this comparison, except with huge uncertainty.
Third, this is the land record. The ocean records are probably an order of magnitude less certain.
Fourth, Satellite records do not fully agree with the instrumental record. This is a major scientific issue. Satellite recordings will certainly improve and give far better records of radiation balance, temperature distributions in the atmosphere, detailed relationships between temperature, urbanisation, geography and the biosphere.
My conclusion is that the BEST study of historical data is of historical interest and we will have much better measurements to understand climate in the near future.
There's no problem with releasing results prior to peer-review, and no problem with telling the press you've done so. The issue is that the contents of the paper are work in progress, tentative, subject to correction, unchecked, etc. So to stand up and start talking about it as if the conclusions were firm, the consensus vindicated, the science settled etc. the moment the paper is released is like going to job interviews and telling them you got an A the day after sitting the exam.
The papers themselves talk about a lot of the things missed out of the analyses or not done yet. There are some features of the results I find 'odd', although I wouldn't like to say they were wrong, and there are other minor statements they make in passing that do appear to be unjustified by the information they present.
It's the same old science-by-press-release story. They release the data and instantly get a big press hoopla going over it, while critics are unable to say anything coherent or well thought out because they've had like half a day. After a few weeks, people start to figure out what was done and possibly what's gone wrong, but by that time the press have lost interest, and the initial impression stands.
What they ought to have done is release the data and the papers, wait two or three weeks to see what the reaction was, and then decide what to tell the press. But I suspect it's a subset of the team who have done this, who have the agenda. A pity - it could have been a great opportunity to mend bridges.
What other examples we have of PR blitz accompanying a scientific paper? Arsenic life? Missing link? Judah's Gospel?
Greenpeace on BEST
http://www.greenpeace.org/international/en/news/Blogs/climate/deniers-exposed-confirmed-again-climate-chang/blog/37463/
LOL.
Thank you for nice information