Monday
Apr082013
by 
Bishop Hill
Bishop Hill Tamsin Edwards versus Jonathan Jones
Apr 8, 2013 Climate: Models Climate: Sceptics BH regulars Tamsin Edwards and Jonathan Jones are facing off in a debate about the usefulness of climate models at the Cheltenham Science Festival on 7 June.
Computer-generated models are used to predict future climates, but how much faith should we put in them to guide future actions? Should we treat their predictions as fact or fiction? With the hot topic of climate change ever current, can we wait to find out? Join climate scientist Tamsin Edwards, sceptic Jonathan Jones and policy adviser Claire Craig. Temperatures could rise in this session…
Details here, although tickets don't go on sale for a couple of weeks.
Reader Comments (59)
Some food for thought. Especially on the construction and testing of models. As an ex-IT person myself, I have to say that looking at the CG1 Harry code, one could only conclude that any models used there were garbage. Shocking code, and clearly CRU had no QA procedures whatsoever. None of that code would have lasted a minute in the commercial world.
Anyway, read on. Tamsin, I'd love to gear your thoughts on what follows, and to know how the Met models match these criteria.
http://blog.squandertwo.net/2006/06/modelling.html
Modelling.
As you may know, I am a computer programmer by trade. As you almost certainly know, lots of scientists these days — especially climatologists — draw conclusions about the real world from computer models. I have therefore compiled this handy list. It's a list of the questions you need to ask any scientist who has used a computer model to reach a conclusion — and I'm not just picking on the climate-change crowd here; they may be the most prominent in the news, but there are lots of other guilty parties out there in all sorts of scientific fields. If a scientist doesn't give confident and reasonable answers to these questions, take their conclusions with a handful of salt.
Who programmed the computer model?
Did the same person do the programming as did the science?
If not, how was the science communicated from the scientist to the programmer? Are you confident that the programmer fully understood the science?
If more than one person programmed the model, do they all have the same background in and approach to programming?
If they have different backgrounds or approaches, what did you do to ensure that their contributions to this project would be compatible and consistent?
What proportion of total programming time was spent on debugging?
Was all the debugging done by the same person?
If not, was there a set of rules governing preferred debugging methods?
If so, are you sure everyone followed said rules to the letter?
Did any of the debugging involve putting in any hacks or workarounds?
If not, could you pull the other one, which has bells on?
Is there any part of the program which just works even though it looks like it probably shouldn't?
Are there any known bugs in the computer hardware or operating system that you run your model on?
If so, what have you done to ensure that none of those bugs affects your model?
What theories did you base the model on?
What proportion of these theories are controversial and what proportion are pretty much proven valid?
What information did you put into the model?
Where did this information come from?
How accurate is the information?
Have you at any point had to use similar types of information from significantly different sources? Have you, for instance, got some temperature data from thermometers and some other temperature data from tree rings?
If so, what have you done to ensure that these different data sources are compatible with each other?
If you've done something to make different data sources compatible, did it involve using any more theories to adjust data? If so, see the previous questions about theories.
Where you couldn't get real-world information, what assumptions did you use?
What is your justification for those assumptions?
Do any other scientists in your field tend to use different assumptions?
Have any of your theories, information, or assumptions led to inaccurate predictions in the past?
If so, why are you still using them?
If they previously led to inaccurate predictions, do you know why?
If you think you know why they led to inaccurate predictions, what else have you done to test them before using them in this model?
How many predictions has your computer model led to that have been verified as accurate in the real world?
How accurate?
Has any other computer model used roughly the same theories, assumptions, and data as yours to give significantly different conclusions?
If so, do you know why the conclusions were different?
How much new information has your computer model given you?
How can we trust a new drug that is marketed? (Long sentence coming up...)
Well, if it was designed, synthesised and tested by the same organisations/people who collected, analysed and reported all the relevant data without rigorous external regulatory oversight, and those same organisations/people stood to gain or lose, financially or career-wise, from the outcome, and some of those same organisations/people were loudly agenda-driven or nakedly political, and their proposed "treatment" (outside of their ostensible skill-set) was equally radical and expensive, and appeared to target a disease that was not yet proven to exist, then...
I would trust it as far as I could kick it.
This is where climate models are today. (And that's before addressing any of the science.) Tamsin Edwards appears eminently trustworthy, but that may not be enough.
Apr 8, 2013 at 7:47 PM | Pharos
Judith Curry is the best scientist working on questions of attribution. She is unique among climate scientists in her willingness to criticize mainstream views. She remains part of the mainstream.
Prof Jones, may I offer this up from Paul Homewood for your delight ?
Met Office’s Private Briefing Document For The Environment Agency
http://notalotofpeopleknowthat.wordpress.com/2013/04/08/met-offices-private-briefing-document-for-the-environment-agency/
@Streetcred
Thank you for drawing attention to this most interesting document.
I look forward to Jonathan Jones's forthcoming interrogation of Tamsin vis-a-vis the Madden-Julian Oscillation (MJO).
(Perhaps)
These comments from Professor Mike Hulme, at the ECLAT-2 Workshop, Helsinki, 14-16 April 1999, are pertinent.
“The climate system, as a complex non-linear dynamic system, is also indeterminate (Shukla, 1998) and even with perfect models and unlimited computing power, for a given forcing scenario a range of future climates will always be simulated. It is for this reason that the Intergovernmental Panel on Climate Change (IPCC) have always adopted the term ‘projection’.”
All the projections are now predictions of course, including the brilliant "retrospective prediction" featured on WUWT: http://wattsupwiththat.com/2013/04/08/on-guemas-et-al-2013-retrospective-prediction-of-the-global-warming-slowdown-in-the-past-decade/
Here are some relevant and recent remarks from a qualified commenter in Sweden:
Lennart Bengsson, described here as one of Sweden's leading climate scientists: http://notrickszone.com/2013/02/03/top-swedish-climate-scientist-lennart-bengtsson-says-warming-not-noticeable-without-meteorologists/
I imagine that that inadequacy in computer models is common ground, although I daresay it has not been appreciated widely in political circles since it is so darned inconvenient for the dominant faction in those circles. A faction that sometimes seems to dearly want a climate crisis to be looming, such are the advantages to them from the prospect.
So, I further imagine the discussion in June will focus on more modest hopes for climate models. Perhaps that they can provide vivid illustrations of hypotheses. Perhaps that developments in climate models will benefit weather forecasting (the 'non-stick frying pans came out of the space programme' argument?). Perhaps that they might help with the development of better theories of the climate system.
But trust them with our wellbeing by leaning on them for policy via the official climate scenarios for the UK (http://ukclimateprojections.defra.gov.uk/)? No thank you. Their track record is poor. But the big problem is that all (e.g. see the 4 'projections' used in the Netherlands: http://www.knmi.nl/climatescenarios/knmi06/index.php) seem to rely on warming projections, and have a fanciful level of detail that could distract the bureaucrats from the highly speculative nature of such exercises. I believe the Dutch at least reserved more freedom for their experts to over-rule model-produced details and make more use of historical and other insights, but I cannot put my hand on a reference for that right now..
A bit of warming would generally be good news, and could be handled readily. Cooling, on the other hand, would not be so nice, nor so easy to deal with. Misleading forecasts may be more harmful than no forecasts at all if they reduce our readiness to deal with climate variability not spanned by the forecasts - cooler weather and shorter growing seasons being two results in particular that spring to mind.
"Can We Trust Climate Models" is the title of the debate. I find it helpful to ask "How do we trust other models?" How can we trust a new drug that is marketed? (Long sentence coming up...)
Well, if it was designed, synthesised and tested by the same organisations/people who collected, analysed and reported all the relevant data without rigorous external regulatory oversight, and those same organisations/people stood to gain or lose, financially or career-wise, from the outcome, and some of those same organisations/people were loudly agenda-driven or nakedly political, and their proposed "treatment" (outside of their ostensible skill-set) was equally radical and expensive, and appeared to target a disease that was not yet proven to exist, then...
I would trust it as far as I could kick it.
This is where climate models are today. (And that's before addressing any of the science.) Tamsin Edwards appears eminently trustworthy, but that may not be enough.
The title of the debate is "Can We Trust Climate Models." That is a fine title for a debate. I will not take cheap shots at it. However, it does suggest that a climate model is something that can be objectively known by any and all who have the expertise to fire it up and run it. That is simply false. A climate model is something that an experienced modeler can ask questions. To judge the quality of answers to questions asked, a modeler must develop a history of model runs, inferences made from them, decisions made, and the value of those decisions in solving the problem at hand. There is no definitive model apart from its systematic use by experts who have specific problems to solve.