Bishop Hill Economist on science
Oct 18, 2013 Journals ;)
The Economist has a fascinating article on the failings of science and peer review which is a nice synthesis of many of the principal critiques that global warming sceptics have been expounding for years. So we hear about Ioannidis's suggestion that most scientific papers are wrong, Fiona Godlee's famous study that showed that peer review was largely a waste of time, and the lack of replication of most studies. It's almost a rewrite of Chapter 15 of The Hockey Stick Illusion.
While it's nice to have one's positions supported by such an august journal, you do have to wonder how the powers that be at the Economist can continue to support revolutionary policy changes on the basis of a system as pathetic as academic peer review. Don't get me wrong - if academics find it useful to peer review each others' work that's OK with me, but we need a much, much higher hurdle before academic papers are deemed worthy of affecting public policy. Independent replication is only the bare minimum required.
Reader Comments (46)
Science (mostly wrong) is not the issue and neither is the peer-review system (almost unfailingly wrong). The problem is the savage political exploitation of science and the peer-review system. Let us not forget that science is based on learning from systematic error while politics is about jumping from one absolute truth to the next.
The only academics who have earned the right to be arbiters of Government Technological Policy are engineers for whom peer review is secondary to real World evidence.
You would not expect railway safety policy to be determined by reference to the output of economists. Similarly, you would not expect energy policy to be determined by people like UEA's Phil Jones, who in the mid 2000s was apparently providing biased data to the major Energy Consultancy contracted to advise on energy infrastructure. As a result, the then government felt it could use windmills alone as a 'virtual power station' when that was never technologically possible.
This has led to a vast increase of energy costs and to get investment in much needed base load power, we have had to cede future control of our new Nuclear facilities to the Chinese State.
Late in my book Credentialed to Destroy: How and Why Education Became a Weapon I get into this issue of trying to develop new economic theories (like what the Club of Rome-affiliated new economics foundation is pushing in UK and its affiliate in the US) to fit with the new "needs" or support economy that "quality learning" globally and the AGW theory for restructuring society all push. Individuals may not all be interdependent as these theories all assert but these theories certainly are all intertwined in the radically different future they are trying to push us towards.
Economics, education, and the law have all become social policy instruments for getting political theories implemented to see what happens. Many of these articles are actually aspirational but being honest about that would certainly negate the likelihood of implementation.
The desired view of the future was actually laid out in the UK book The Spirit Level that is now being turned into a film to drive the more visually oriented to take action. Too bad for us the visions are straight out of Marx, who is certainly making a comeback in the 21st century through his human development theory and vision of the progression of history. Advancing those theories without using the M word has a lot to do with who gets published and who gets criticized.
"much higher hurdle" ??
It would be very interesting to read some suggestions on what that hurdle might be and how, and by whom, provided. I cannot imagine anything that would work while not protecting us from some of the past inspirations which have turned out reliable. Maybe one of the current practitioners will have an idea.
Good article interesting the suggested solution basically asks for improved statistic skills and auditing.
It touches on peer review but I think one aspect of PR that could have been further explored is the fact at one time PR could reasonably be accepted as a tool for minimizing bad science but its application has now gradually morphed to become a gateway for controlling "truth" and therefore power.
The recent sting on Open Access papers by Science provoking reprisals against paid journals shows PR becoming a battleground for controlling power I think, rather than the ostensibly benign proposed motives by both sides for quality versus access .
As the intertwining of "peer reviewed" science with politics is a deadly combination (e.g. eugenics, Lysenko): the science will get the blame and loss of reputation,; politicians stay, an obvious solution to the problem seems te be: When politicians want to use the "peer reviewed" scientific results to guide their policy decisions and agendas, then by law, repetitions by financially independent scientists (not)confirming the results intended to be a part of a policy decision, should be brought forward in the discussions.
Good to know, too, that the Heir to the Throne is still trotting out the 'catastrophic climate change' mantra - clearly if he really does talk to his plants, you would have thought that by now they would have told him that CO2 is nothing but beneficial to them...
@oebele bruinsma: a good idea because as the exaggeration and falsification levels increase, the probability of that being proven also increases. This was of course the original reason for peer review.......
[Nowadays, you have a corrupt group doing the peer review.]
"... we need a much, much higher hurdle before academic papers are deemed worthy of affecting public policy. Independent replication is only the bare minimum required".
Exactly!
Perhaps you should write to them asking for royalties.
Or at least that they should ask you to write it next time they want to discover that you were right all along.
Folks - read 'The Button Collector' over on Wattsupwiththat....
Thought-provoking..!
Well put Andrew, after all - what has science ever done for us?
Always amused by this pose. Really, scientists should be seen as selfless philanthropic heroes who made our word today?
This pose implicitly takes all the credit for the actual applications of technology that often proceed the later scientific studies enabled by their wealth generation, and/or the capitalist motivated industrial discoveries that provided the comfort of modern society.
The public face of scientists today is they have nothing to offer “us” but censure on our lifestyles and all these prior things "done for us".
That article is now up for discussion at Judith Curry's blog - "Trust, and don't bother to verify"
I think it makes some good points:
"Careerism also encourages exaggeration"
"The most striking findings have the greatest chance of making it onto the page"
but overall, the article is too sweeping in its criticism of science, and ironically is an example of exaggeration and striking findings getting published.
Some interesting stuff can be found here:
http://retractionwatch.wordpress.com/
Oct 18, 2013 at 1:34 PM | Doug McNeall
The aqueduct?
:-)
Hmmm I think the Aqueduct owes more to Engineers than scientists :-)
@Richard Betts and the sanitation ;)
I'm with Paul on this one. The idea that (for example) peer review is worthless is surely just as dangerous as the idea that it is perfect.
I'd be interested to hear about Andrew's standards for the use of science in policy, whether there might be any asymmetries in their application across fields, and to discuss what the practical barriers to their adoption might be.
@Leopard, that is some pretty heroic extrapolation from my short (and perhaps slightly flip) comment.
Cagwaduct: where back flow causes the level to rise until it overflows.
Re: Doug McNeall
> Well put Andrew, after all - what has science ever done for us?
Science has done a lot, scientists on the other hand ...
According to the US Bureau of Labor Statistics 4.3% of all employment counted by OES is in Science and Engineering (2008). This excludes a further 1.5% that are employed in support occupations such as technicians, programmers and Science and Engineering managers.
With this many scientists you would expect a lot more and faster progress, unless, of course, a lot of what they produce is useless or simply wrong.
If the Editor of the Economist pops in to BH to see what's being said about his "august journal", I'd like to address these comments to him.
Dear Ed, I cancelled my Economist subscription when you supported the invasion of Iraq. Since then, when periodically tempted to forgive you, your continued support of the Global Warming Myth keeps my wallet in my pocket. Come the day that an editorial states "we were wrong about Iraq and also about Global Warmery", I shall be happy to again subscribe.
@Oct 18, 2013 at 2:51 PM | Unregistered CommenterDoug McNeall
Extrapolation? You started it.
Thanks for the opportunity of reminding me of all the other times I hear variations of that specious extrapolation, helped motivate my extrapolation to be "heroic" ;)
@Richard Betts: I doubt that you will find many visitors to BH who are "anti-science".
The essential distinction you need to understand is between the process of science and the fallible human beings who practice it.
("Fallible"... I hope that you will see the restraint in that moderate word. I'm so glad that I resisted the temptation to rail against the rent-seeking gravy-train integrity-free apocalypse-merchants peddling the Global Warming TM hysteria to the despoilation of Britain's beauty spots, the energy-impoverishment of those citizens least able to afford it and the attempt to - hah! - decarbonise and deindustrialise the economy.)
Chaucer had you lot figured 600 years ago in his Miller's Tale.
Brent
The editor of the Economist is John Micklethwait.
You may be thinking of Edward Lucas, a journalist there who is IIRC married to Cristina Odone.
Peer review is essential, its just that it is not what some people think it (a) is and (b) should be. For one thing it is not an independent scientist replicating the results. It also doesn't generally involve going through a paper with a fine toothcombe - there isn't time. Rather it is a check that the work is plasuible, has been conducted in a suitably scientific manner, has enough detail to allow others to repeat the work and has conclusions that are justified by the results. Lastly - is it os suffiecient standard and relevance for the particular journal it has been submitted to.
In my field of chemistry, people indirectly repeat results when they try to syntheize reported compounds that they are interested in. You find out pretty quickly if things are not working, and indeed this is a valuable check on papers, but it's not peer review. As far as I am aware, this isn't going to happen in much of climatology, and that is one of the big problems.
There are others - I suspect that the lack of doubleness in peer review is a barrier. Boggins, well known in the field, will often find it easier to get work published than mr just starting my career. As a British scientist at a British Uni we find it harder to get work into US high impact journals than US scientists reporting similar work. This is something which i feel bedevils Climate science.
Justice (by the way, I have long been curious about the identity of the Rinka for whom you seek justice, but that's another matter),
One key quality I used to admire about The Economist was its uncompromising rationality. With the exception of (and quite properly) the Arts sections, the 'house ethic' was that all articles were evidence-based. The anonymity of articles was a tacit statement of that rational approach. It really didn't matter which journo wrote a piece because the house ethic, and the commitment to fact, was so strong.
Compare that ethos (sadly now lost) with pre-climatology science where, say, F=ma or E=mc^2 is true regardless of who says it or writes it; the truth is greater than the individual who conveys it.
(Richard Betts - are you following this?)
Turbo,
As a practitioner of a hard science, Chemistry, do you have any sense that your delinquent little brother Climatography (IMO Mann & Co do not rate an 'ology') is in danger of bringing the wider body of science into disrepute?
Seems to me that the public - including members of my family - accord to scientists (Richard Betts - are you following this?) a status once reserved for religious authority figures. Joe Public can scarce imagine that Sir Mark Wolpert - an authority in his field - is as clueless as the next man outside of it.
Scientists - frail and fallible human beings - should chant every morning and every night "Nullius in verba" and when speaking to the public, "includingus mine-us"! (Richard..... oh, what's the point?)
Well bishop, how about a concrete proposal!
I think it is spiffing that the world realises that most professions are rubbish and not *just* the bankers. In this basis I am more than willing to join the effort to give scientists a collective shoeing. Anyone fancy a joining me for plate of phlogiston first?
It is a political, not a scientific problem - how to prevent politicians from receiving bad advice.
A few years ago I did a study with Bruce McCullough (an econometrician who specializes in replication work) on the need for due diligence when scientific research is used in policy formation.
http://www.rossmckitrick.com/uploads/4/8/0/8/4808045/thecaseforduediligence_fiedition.pdf
We mainly focused on the problem of non-reproducible work in economics, but added examples from lots of other disciplines as well, including an episode in paleoclimatology that was getting a bit of attention at the time. We concluded with a specific proposal for how academic publications should be vetted before being considered sound enough to base policy on. (Sorry, this is a British website; that should read: before being considered sufficiently sound on which to base policy.)
One of the episodes we summarized concerned the Harvard 6 Cities study of air pollution and health effects. Yesterday I was giving expert witness testimony at an Environmental Review Tribunal hearing and underwent a 3 hour cross-examination by a team of government lawyers. I think we spent nearly an hour on that paper alone, even though it was peripheral to my evidence. It was obvious the lawyers had no idea about any of the controversy that surrounded that paper, the HEI reanalysis much later, etc., and unfortunately because of the structure of the cross-examination, none of that material was germane. The lawyers were prepared to take, not merely the findings of the 6 Cities paper at face value, but their interpretation of the findings (which was a bit extrapolated, shall we say).
The point is that people in government make very motivated uses of research, and are happy to exploit any ambiguity as needed to provide cover for a position they had decided on on entirely different grounds. This is not a situation that can be addressed by monkeying with the journal peer review process, it has to be fixed by creating an audit process in government itself so as to qualify scientific information according to objective criteria, chiefly whether the data and code exist to allow independent replication.
Interesting article. On my list I've put it just above The Decline Effect. Science is just as dysfunctional as any other Human activity, and the people who promote it just as blind to the fact.
Rinka was Norman Scott's dog. Jeremy Thorpe allegedly hired a hit man to rub out Norman Scott, but after shooting the dog his gun jammed, and Norman lived to tell the tale and end Thorpe's unremarkable career.
I signed up to LibDemVoice years ago and twitted them gently about the rubbish quality of their leading figures, who seem to consist disproportionately of sexual perverts, drunks, pompous old fools, grossly conflicted crooks, and people like Cyril Smith who were in fact all four.
None of my handles lasted more than a couple of posts before getting banned, so after a while, I just changed my handle to Justice4Rinka and posted bland inanities. The idea was simply to mention Rinka as often as possible on their website really.
Eventually they worked it out and banned me again, but it took them a long time.
It's a disturbing thought that but for fate, we might now have Mark Oaten (coprophile), Charles Kennedy (alcoholic), and Chris Huhne (proven liar) all in government.
For a moment there I thought Ross's point was that lawyers build on case history, and that very body of knowledge acquires its own truth - entirely legitimate in the matter of law. But he then went on to observe that politicians, with their preconceptions, pick and choose evidence to support their original position. The word 'truth' means two different things to the Arts and to the Sciences.
And JFerguson's point is also very pertinent: "It is a political, not a scientific problem - how to prevent politicians from receiving bad advice?" This one is enough to drive one bonkers... one can hardly blame a politician for accepting scientific advice; the contrary - to declare, "I, unqualified politician, know better than the experts with letters after their names" is indefensible, although the UK Govt recently sacked a Drugs Czar whose advice they didn't like.
With those 3 points - lawyer's adaptable truth, politicians' pre-programmed truth and the new cult of scientific truth-because-we-say-so, where does that leave us? My tentative view is that in matters of science, dodgy science advisors can easily pervert public policy. WIthout integrity - and this includes a frank statement of the limits of their individual knowledge - we might as well have Tony Soprano giving policy advice.
Solution? Other than an eventual show trial of the DECC CSAs and GCSAs (the guilty men), only the "collective shoeing" of the scientific profession will do it. Or - just maybe - it's time for Chemistry and Physics to energetically dismiss Climatography as pseudoscience, to disassociate themselves from the scam even at the risk of attack by green extremists. Edmund Burke said, "the only thing necessary for the triumph of evil is that good men stand by and do nothing". C'mon Chemistry, grow a pair!
Economics is not science.
Economics is better associated with, and more closely related to, ideology.
Turbotubbs: can you trust a scientist who cannot even get their English right? There is no such thing as a toothcombe, fine or otherwise; you probably mean fine-tooth (or –toothed) comb (even “combe” is a misspelling). Spellchecker or not notwithstanding, you do present a good argument – people (including me) do seem to equate “peer reviewed” with “peer verified”; in other words, the writer’s peers have performed the same experiment and arrived at the same result. However, “peer review” does seem to be more of a question of, “Yes, the writer is my mate, and I trust what he says, and it looks good, and there’s lots and lots of numbers, and there is no argument from me, and I want to keep my job…” (look at the flak Chris de Freitas received from deviating from that principle) “…and, where’s my cheque?”
We should all remember that scientists are also human, with all the flaws that humans have; it is just that there seem to be some of the climate scientists who are convinced that they are infallible, and how dare anyone question their pronouncements!
"It is a political, not a scientific problem - how to prevent politicians from receiving bad advice."
Oct 18, 2013 at 5:24 PM | Registered Commenterjferguson
____________________________
The current issue of Oxford Today, a magazine for Oxford alumni, contains an article about Margaret Thatcher as the first UK Prime Minister with a science degree. It ends as follows:
“One of the major reasons why climate change debate has never been as partisan an issue in the UK as in the US, was Thatcher’s very early embrace of the science in the late 1980s, which convinced her of the case for action. The speeches she made in 1998 to the Royal Society and in 1989 to the UN General Assembly on the emerging science and the need for action have stood the test of time very well.”
To savour the full piquancy of this passage, you need to know that its author is a PPE graduate and former civil servant at No.10. Nothing is said about Thatcher’s later change of mind; it is perhaps surprising that an impartial civil servant did not feel it necessary to mention this.
In the same issue, there is an article lamenting the lack of scientifically qualified people in the upper ranks of the civil service. So far as climate science is concerned, HMG is evidently a reality-free zone. What a tragedy that Thatcher allowed herself to be duped into promoting the world’s greatest scientific fraud; she was the one person who might just possibly have stopped it in its tracks.
Here is what the UK parliament think peer review is:
and
Peer review is a multi-billion pound industry in the UK.
The Economist isn't the only magazine that has an article on the failings of science this week. The New Scientist (19 October 2013) has article about neuroscience and an editorial, both on this subject. Both mention the paper by John Ioannidis. The editorial discusses the current situation and states"
Things are by no means perfect now. Scientists are under immense pressure to make discoveries, so negative findings often go unreported, experiments are rarely replicated and data is often "tortured until it confesses". But - thanks in no small part to Ioannidis's brutal honesty - all of those issues are now out in the open and science is working to address them.
Presumably the answer to these problems is for all sciences to become more like climatology. After all, the Royal Society, its foreign counterparts, and the various official enquiries into Climategate, have all shown that climatology is completely free of any of the problems that the New Scientist, like the Economist, admits afflict other sciences.
IMO that question was phrased incorrectly. It should say: "What have scientists ever done for us?"
According to Compact Oxford English Dictionary (currently crushing my knees) the word "scientist" was first coined in or just before 1834 by the British Association for the Advancement of Science after rejecting philosophers and savants.
Therefore there were no scientists prior to 1834.
Most of the people who engaged with science in the rest of the 19th century were builders and would more correctly be considered engineers. The scientists were off pondering the nature of important things like aether and phlogiston.
Maybe Darwin was a scientist.
Therefore scientists have been instumental in bringing us (or at least introducing us) to everything after steampunk.
In my opinion the question is "Why are the scientists trying to do for us?"
"Therefore there were no scientists prior to 1834."
Quoted from Johann Peter Eckermann's conversation with Goethe, Feb. 1, 1827
"'This is not to be wondered at,' said Goethe; 'such people continue in error because they are indebted to it for their existence. They would have to learn everything over again, and that would be very inconvenient.'
"'But,' said I, 'how can their experiments prove the truth when the basis for their evaluation is false?'
"'They do not prove the truth,' said Goethe, 'nor is such the intention; the only point with these professors is to prove their own opinion. On this account, they conceal all experiments that would reveal the truth and show their doctrine untenable. Then the scholars -- what do they care for truth? They, like the rest, are perfectly satisfied if they can prate away empirically; that is the whole matter.'"
What's new?
NoIdea
But...but....but Lewadowsky has settled the science: it is only wicked right wing denialist scum who are doubting science these days.
I like the idea of scientific journals circulating a lot of ideas, some good, some bad, trying to avoid arbitrary gate-keeping and in other ways improving standards compared to today, encouraging peers to show off to each other, letting the chips fall where they may. On the other hand, this activity should probably have as little as possible to do with political decision-making. Only rare, probably expensive research, that has been replicated, should be taken seriously as the basis to build windmills and such.
I don't know whether politicians have corrupted scientists or vice-versa; they bring out the worst in each other. Aristotle would predict that politics always wins in the end; boomer scientists want to be seen as good citizens, saving Bambi etc.
One reason the IPCC stays alive, a bit like Frankenstein's monster ("you got the brain from what patient? A.B. Normal?"), is that its products are coated with a respectable grey dust called "peer-reviewed literature." (This is one case where dust seems to have caused heat rather than cooling--sorry, I think I'm responding to a thread at Judy Curry's site). I like the Bish's suggestion a while ago that we should all have a nice clear list of the conclusions of AR4 and AR5 side by side. There used to be a preferred sensitivity estimate of 4 degrees C per century; now there is not one, probably because the literature (more extensive than before) would point to 2 degrees or less--doesn't fit the narrative. By their own reckoning (table 12.4), there used to be 9 impending catastrophes, now there is one--and that one (Arctic ice) is regional, not global. AR 4 says there are more weather disasters than before because of climate change and that they are costing more. But AR 5 confirms that increased disaster costs are not due to global warming but tosocietal factors, such as increased development in hurricane-prone areas.
It looks like the warmists are going to try to stick to AR4 and pretend AR5 isn't happening--except for the 95% line, which is just for the children, er, decision-makers. So before we bid a fond farewell to AR4, let's develop a list of statements that have been shown to be false and misleading with further investigation (perhaps referred to as "BS") and those that were known to be false or deeply misleading at the time (properly called "Double BS" or "BS Squared").
The issue of the Himalayas (BSS) has been beaten like a drum. Here are some others:
- The summary of temperature predictions based on models runs warmer than AR 1,2, or 3; that's why the new summary tries to drop AR4 on this issue (BS).
- A section about agriculture in northern Africa says global warming and normal climate variability could reduce crop yields. But it gets oversimplified in later summaries so that lower projected crop yields are blamed solely on climate change. (BSS)
- Speculation as to whether there will be fewer people with potable water, or more people, overall, skews in the opposite way to the published source that is cited (paper discussed winners as well as losers, shows more net winners for a net improvement overall; IPCC makes no mention of winners). (BSS)
- The second-last version, which reviewers worked on, cited possible problems with drought and crop production, including tea; the last version included changes that the famous peer reviewers had no say in, making everything seem worse and more certain. (BSS)
- One peer-reviewed study of a small section of Amazon rain forest, which had suffered extensive logging, fire and perhaps other disasters, was magnified into a claim that the entire rain forest was at risk. (BS or BSS, or both)
The problem isn't so much what appears in the journals as the misrepresentation of the papers by University PR agencies combined with scientists just making stuff up for media consumption, based on no papers, data or even a plausible theory.
However it is a bit rich that economists have the cheek to criticize anyone else despite their own dismal failures. I'd like to see the same critiques from the economist on how easy it is for new and daft economics policy ideas to take root despite having zero real world testing and lots of outsiders in the real world saying they will make things worse.
JamesG - which "new and daft economic policy ideas" are you referring to?
It's a genuine question. AFAIK, pretty much every economic policy idea has been tried at one time or another. True, the situations to which they are applied may be new. But there are not a lot of genuinely new "ideas" in economic policy.