BBC R4: "Everything we know is wrong"

BBC Radio 4 produced an amazing programme this week on the problems with scientific research. Everything that has been said by sceptics about climate science was here - they even describe a 'decline effect' - how delightfully ironic. Here is the programme blurb:

Every day the newspapers carry stories of new scientific findings. There are 15 million scientists worldwide all trying to get their research published. But a disturbing fact appears if you look closely: as time goes by, many scientific findings seem to become less true than we thought. It's called the "decline effect" - and some findings even dwindle away to zero.

Click to read more ...

Finding fraud in scientific papers

Judith Curry tweets a link to a fascinating report of a prototype technique for unearthing fraudulent scientific papers:

"The analysis revealed that Stapel’s fraudulent papers contained linguistic changes in science-related discourse dimensions, including more terms pertaining to methods, investigation, and certainty than his genuine papers. His writing style also matched patterns in other deceptive language, including fewer adjectives in fraudulent publications relative to genuine publications," the authors write.

Stapel tended to fortify his methods section with extra description and employ words like ‘‘profoundly,’’ ‘‘extremely,’’ and ‘‘considerably’’ to make his results sound more convincing and dramatic. At the same time, he also used fewer terms that might downplay significance, such as "less," "somewhat," and "merely."

Someone could have fun with this couldn't they!

Parliamentary links day

Updated on Jun 24, 2014 by Bishop Hill

Updated on Jun 24, 2014 by Bishop Hill

The House of Commons is having a "links day" in which MPs will get together with scientists to discuss the issue of trust in science. Mark Walport and Paul Nurse will be speaking. I've been following the tweets on the #linksday2014 hashtag and they are a mixed bunch so far.

For example, we learn that Nicola Gulley, the editorial director of the Institute of Physics opined that:

...peer review key to maintaining trust in science. No crisis but a lack of understanding of this process. 

You can see why someone working in the peer-reviewed journal sector might be keen on peer reviewed science, but for many readers at BH and many others uninvolved with the climate debate, peer review - its ineffectiveness, the superficial aura of "correctness" it gives, and the problem of gatekeeping - are the source of mistrust in science not a solution to it.

Click to read more ...

Simon Singh on peer review

I was intrigued to see Simon Singh retweeting a letter to the Times criticising the peer review system. Penned by Professor Tony Waldron of UCL, it described peer review as being a system in which one group of scientists does its best to stop another group from publishing. I'm sure plenty of readers will have no dispute with this.

Given Singh also has great enthusiasm for mainstream positions on climate change and a tendency to invoke the peer reviewed literature and the IPCC in his support, I wondered if these two positions weren't a bit inconsistent.

Quote of the day, Nature edition

My default position toward Nature...at least for earth and environmental science papers, has shifted from innocent until proven guilty, to roughly the opposite. I just don’t believe what they claim until I’ve read the paper involved closely, and since I don’t have time to do that, that means I basically don’t accept what they claim. I’ve just seen too much bad science and I don’t trust them to be fully objective and place scientific veracity over hype and headline. Sorry.

Jim Bouldin, an ecologist from UC Davis

The opinions of experts

Stephan Lewandowsky has launched the next round of the Recursive Fury saga, quoting an excerpt from the report of the expert panel that Frontiers commissioned to look into the ethical and legal issues surrounding the paper. The report says that there are divided opinions in the field as to whether analysing blog comments for a scientific paper would require informed consent, but seems to end up saying that the Fury authors' use of such comments was probably kosher.

I think I probably agree with this. I can't really see any objection to studying public blog comments. But I'm not sure that this doesn't miss the key objection to the Fury paper, namely that the authors published what amounted to diagnoses of the (alleged) psychological pathologies of identifiable individuals without their consent. I can see no way in which this could ever be acceptable practice for a reputable journal.

More from Markram

Following on from Frontiers' recent statement about the retraction of the Lewandowsky paper, the journal's editor Henry Markram has left a comment giving his personal views of the affair (H/T Paul Matthews):

My own personal opinion: The authors of the retracted paper and their followers are doing the climate change crisis a tragic disservice by attacking people personally and saying that it is ethically ok to identify them in a scientific study. They made a monumental mistake, refused to fix it and that rightfully disqualified the study. The planet is headed for a cliff and the scientific evidence for climate change is way past a debate, in my opinion. Why even debate this with contrarians? If scientists think there is a debate, then why not debate this scientifically? Why help the ostriches of society (always are) keep their heads in the sand? Why not focus even more on the science of climate change? Why not develop potential scenarios so that society can get prepared? Is that not what scientists do? Does anyone really believe that a public lynching will help advance anything? Who comes off as the biggest nutter? Activism that abuses science as a weapon is just not helpful at a time of crisis.

There is at least some common ground.

Reddit, dislikedit, deletedit

Stephan Lewandowsky is doing a two-day question and answer session at Reddit. The first day's questions were fired off in the great man's direction today with some eye-opening results.

As Jo Nova reports, Lew having responded to a Richard Tol question about data availability by saying that he was all in favour of it, Barry Woods decided to ask about Lew's own data, quoting the University of Western Australia's response stonewalling of an earlier request.

At which point Reddit decided to delete the comment.

Reddit, dislikedit, deletedit.

More Lewdness

The Lewandowsky affair shows no signs of dying down. Following Frontiers' decision to kill off the 'Recursive Fury' paper once and for all, Lewandowsky has responded by setting out his thoughts in a post at his own blog.

In this version of events the original retraction notice was agreed between the legal teams of Frontiers and the authors (is it normal to get the lawyers involved for this kind of thing?) and Lewandowsky is highlighting discrepancies between what was said in that document and what Frontiers claimed in its subsequent clarification notice.

Click to read more ...

The Lew letters

Steve McIntyre and Barry Woods have published their correspondence relating to Lewandowsky's paper- Steve's complaint to Frontiers in Psychology and the University of Western Australia about breaches of the various ethical codes and Barry's attempts to get hold of the underlying data from the University of Western Australia.

This correspondence and the failure of the university to act upon any of it suggests that the problem at UWA is not restricted to one rogue researcher. The ethical failures seem to go right to the top.

Update on Apr 5, 2014 by Bishop Hill

Anthony Watts has posted up his correspondence too.

The final Frontiers

Frontiers, the journal that published and subsequently retracted Lewandowsky's notorious 'Recursive Fury' paper has issued a statement in an apparent attempt to draw a line under the affair. It at least seems to have put an end to suggestions that threats of libel action had anything to do with their decision. In fact the statement could be construed as "throwing Lewandowsky under the bus".

Retraction of Recursive Fury: A Statement

(Lausanne, Switzerland) – There has been a series of media reports concerning the recent retraction of the paper Recursive Fury: Conspiracist ideation in the blogosphere in response to research on conspiracist ideation, originally published on 18 March 2013 in Frontiers in Psychology. Until now, our policy has been to handle this matter with discretion out of consideration for all those concerned. But given the extent of the media coverage – largely based on misunderstanding – Frontiers would now like to better clarify the context behind the retraction.

Click to read more ...

Flushed away

Via Ben Pile we learn that the Lew Paper - the 'Recursive Fury' one, about reactions to the bonkers conspiracy theorists one - has been retracted, or is about to be. It seems that a Dana Nuccitelli post went up at Skeptical Science announcing the paper's end an hour or so ago. The post has now been removed from public view, although Google's cache enables us to see it in all its glory.

...nobody likes being called a conspiracy theorist, and thus climate contrarians really didn't appreciate Recursive Fury.  Very soon after its publication, the journal Frontiers was receiving letters from contrarians threatening libel lawsuits.  In late March 2013, the journal decided to "provisionally remove the link to the article while these issues are investigated."  The paper was in limbo for nearly a full year until Frontiers finally caved to these threats.

Click to read more ...

The peer review game

There is an interesting letter in Nature this week. In-Uck Park of the University of Bristol and his colleagues have adopted something of a game-theoretic approach to try to understand aspects of the peer review process.

The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists’ decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers’ behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.

Click to read more ...

One small step for Science

Marcia McNutt, from June last year the editor in chief of Science, has issued a new reproducibility policy for the journal.

Science advances on a foundation of trusted discoveries. Reproducing an experiment is one important approach that scientists use to gain confidence in their conclusions. Recently, the scientific community was shaken by reports that a troubling proportion of peer-reviewed preclinical studies are not reproducible. Because confidence in results is of paramount importance to the broad scientific community, we are announcing new initiatives to increase confidence in the studies published in Science. For preclinical studies (one of the targets of recent concern), we will be adopting recommendations of the U.S. National Institute of Neurological Disorders and Stroke (NINDS) for increasing transparency.* Authors will indicate whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment. These criteria will be included in our author guidelines.

This is a start I suppose. I can't see anything about availability of data and code, which is always going to be the starting point for reproducibility. Still, every little helps.

Reader Lance Wallace sends this further excerpt:

Because reviewers who are chosen for their expertise in subject matter may not be authorities in statistics as well, statistical errors in manuscripts may slip through. For that reason…we are adding new members to our Board of Reviewing Editors from the statistical community to ensure that manuscripts receive appropriate scrutiny in their methods of data analysis.

Which is definitely a win.

McIntyre's mirror image

This article in the Guardian about an MSc student who uncovered a major flaw in a headline grabbing psychology paper is amazingly reminscent of Steve McIntyre's story: the amateur sleuth, the mathematically illiterate academics, the unwillingness to admit error; it's all there.

"Not many psychologists are very good at maths," says Brown. "Not many psychologists are even good at the maths and statistics you have to do as a psychologist. Typically you'll have a couple of people in the department who understand it. Most psychologists are not capable of organising a quantitative study. A lot of people can get a PhD in psychology without having those things at their fingertips. And that's the stuff you're meant to know. Losada's maths were of the kind you're not meant to encounter in psychology. The maths you need to understand the Losada system is hard but the maths you need to understand that this cannot possibly be true is relatively straightforward."