Bishop Hill Tol responds to Ackerman
Oct 28, 2011 A couple of days ago I posted an article by Frank Ackerman in which he defended his estimates of the "social cost" of carbon and made some criticisms of Richard Tol's work. This guest post by Tol is a response to Ackerman's piece.
Over the last 20 years, I have developed (and later co-developed) an integrated assessment model of climate change called FUND. Model code and documentation used to be available to anyone on request. It can now be freely downloaded. Reproducibility and transparency are cornerstones of scientific inquiry.
Some modelers prefer to keep their code private. There are a number of reasons for this. One reason is the potential for abuse. Someone may borrow your model, do something inappropriate or silly with it, and use the results to embarrass you.
I have borrowed other people’s models. I have found bugs in their codes, or what seemed to be bugs. I always discuss this with the modeler in question. If there really was an error – more often it is a misunderstanding on the part of the outsider – I left it to the modeler to correct this and whatever results that were affected.
Colleagues have treated me, my code and my data in the same way. I have had to publish two errata.
Frank Ackerman borrowed our model. He discovered that in one equation there may be a division by zero. He contacted my co-developer, David Anthoff, on 16 December 2010. In the following email exchange, David introduced Mr Ackerman to our standard diagnostic procedures and explained to him that our results are not affected. David added bespoke diagnostic tests on Mr Ackerman’s request and again found immaterial effects.
Much to our surprise, Mr Ackerman claimed in a blog post on 15 March 2011 that there is an error in FUND. He recently repeated this claim (and again, and again). In an email dated 17 March 2011, Mr Ackerman admits that he used different methods than we do to work around the division by zero; and that if he uses these methods, there is a substantial error in the estimates of the social cost of carbon.
The computational error, therefore, is Mr Ackerman’s.
I am reminded of Phil Jones’ “Why should I give information to you when all you want to do is find something wrong with it?” although in this case it is more like “Why should I give you my code when all you want to do is introduce a bug and blame it on us?”
Nonetheless, FUND will remain in the public domain.
Reader Comments (21)
Hi Richard
I recommend you have a very strict naming convention for versions / configurations of your model. For example, FUND1.0 would be the model as you set it up, which you know works. If anyone else takes it on and modifies it, they should not be allowed to call it FUND1.0 - and they shouldn't call it FUND1.1 either unless their changes have been approved and checked by you. They should say they are using a modified version of FUND1.0, or give it their own sub-version name (eg: FUND1.0_A or something) to be clear about what comes from you and what comes from them.
@Richard B
We use MS Visual Studio with automatic model versioning, archiving of change logs, rollback, and watermarks. A forensic analysis would reveal that Mr Ackerman has created his own branch and did unspeakable things to the code.
Unfortunately, the technology is not yet so far advanced that blog posts bear the model's watermark.
I downloaded it and gave it a quick look over (my IDE had to upgrade the project to a later version, but there were no conflicts or errors). Didn't have time to examine it in great detail, but I did run it with no command line parameters just to make sure it built and ran.
May take a more detailed look later.
This is going to be an increasing problem as source code goes into the public domain.
You have my sympathy. I had a similar problem when some idiot technician altered some of my code.
I have encountered it, although in a less public form, when an idiot technician decided to alter some code of mine.
I think the solution may be to have a contract in which you give the executable files and the source code and put in a strict condition that it the source is modified, it is no longer your model and this has to be disclosed by the recipient.
Looking at your code (briefly) I could see plenty of scope for error in modifying it.
The trouble with releasing code is you're damned if you do and you're damned if you don't.
rc:
I would not use the phrasing you suggest since lawyers could use such language to question your ownership rights. Rather I would say that a statement recognizing that when changes are made to the code it can compromise the integrity of the model and the results and such results are no longer comparable to earlier results unless acknowledged to be so by the original developers.
In the third comment, Mr Ackerman admits to changing the code:
http://triplecrisis.com/for-whom-the-blog-tols/
He writes that he changed one line of the code. In an email, dated 17 March 2011, he mentions more extensive he made to the code. I cannot release that email without violating his legal right to privacy.
May I raise an impudent impish but serious challenge to the apparent 'taken for granted' notion of 'the "social cost" of carbon. How about the social benefits of carbon? By carbon, I mean the carbon of fossil petroleum compounds, the exploitation of which has driven a revolutionary transformation in social conditions from 300 years ago?
I make no apology for my small part in UK reserve additions, and still consider them a net benefit, however unfashionable that prevailing, politically incorrect, sentiment may be.
Richard Tol
Well, I am not so sure about the violating his legal right to privacy given he has gone public with the comments he has made. But I do know that you can sue the bastard.
What most people miss is that there is something called "Moral Rights" which you as the creator of a work, in this case the computer program, still own. In fact, unlike the literary copyright which you apparently gave away by making it public domain, you cannot give away moral rights, particularly given that you are living and working in Europe. (In the US moral rights are not well enforced, unlike the EU.)
Moral rights pertain to not only the right of paternity or attribution, that is the recognition of the fact you created it and not somebody else, but also the integrity of the work. It is this latter point of interest.
Even as the owner of a work of art, I cannot deface it and then display it as the work of the artist without his approval. The situation here is the same. He took your copyrighted work (that is automatic unless you did it as a work for hire) and defaced it (changed code) and then said that it was your work.
Have fun. It will cost money to sue and hire a lawyer, but it is clearly the case.
However, you can also try a threatening letter from a lawyer or solicitor. You might force him to openly admit his changes and than his statements are based on defaced code. It is surprising what a stiff letter from a lawyer can accomplish. That might be worth the couple hundred euro it would cost.
@Pharos
FUND also computes what you call the benefits of carbon -- that is, there is a cost to reducing greenhouse gas emissions. The model also seeks to balance the costs of emission reduction with its benefits. That's a discussion for another day, though.
Richard T (or Frank Ackerman if you are still reading)
It's really interesting reading about the details of the model - it's very different to the GCMs and land surface models that I work with, and I've not really worked with IAMs.
Putting aside the argument about technical errors, may I ask a few questions about Frank's uncertainty analysis (which I agree is a good thing to attempt - it is indeed important to try to explore the implications of the large uncertainties in projected climate change).
First, did the uncertainty analysis really only consider the median and 95th percentile of climate sensitivity, and not the 5th percentile - this would seem to imply that only the upper half of the distribution is being examined. Maybe I've misunderstood and "95th percentile" was shorthand for 5th and 95th percentile - please can you clarify?
Second, neither of the blog posts nor the model documentation mention precipitation, it all seems to be about temperature, with agricultural productivity (for example) being empirically linked to regional temperature change, using relationships calibrated from the literature. I'd have expected an uncertainty analysis to explore uncertainties in regional precipitation - surely this will be a key influence on productivity, and the uncertainty is first-order so needs to be examined. Again, have I missed something - was this included in the Ackerman analysis?
Thirdly it would be good to further details of the post-1996 updates to the agricultural sensitivity parametrisation (this seems to be mainly about extreme temperatures). What are the references for the more recent work that suggests the original calibrations are unreliable?
Finally did the CO2 physiological effect remained unchanged in the Ackerman version of the model? This is a critical process, any many studies explore the uncertainties in it's effects rather superficially by either having it switched on or off (I do that myself in sensitivity studies!). It would be good to know whether the update to the agricultural calibrations included anything with the CO2 parameter or not.
I'm hoping Frank is still around to answer these questions, but if not, maybe Richard can. I'd really like to understand more about this model and what's being done with it by various people as we start working on the AR5 WG2 First Order Drafts - I'm keen to understand more about the traceability between the IAM's used for these high-level economic impacts assessments (WG2 territory) and the climate projections in WG1.
@Richard B
1. Ackerman and Stanton indeed only considers the mean and 95%ile. See
http://www.economics-ejournal.org/economics/discussionpapers/2011-40
In our analyses, we consider the entire distribution of all uncertain parameters. See
http://iopscience.iop.org/1748-9326/4/2/024002
2. All climate variables are scaled with the global mean temperature.
3. Agricultural impacts are calibrated to models that include international trade. Those models lag behind the latest insights into yields.
4. The impact of CO2 fertilization has been adjusted downwards in the light of recent papers.
Richard T
Thanks - interesting.
We'd better keep in touch on this, especially (4) - there were some very important comments on our ZOD on this topic.
I just had a browse through the code and was surprised by the sheer number of things that turn out to be a simple function of world temperature.
Is there anyone outside the club that takes this nonsense seriously? I mean besides the ever-faithful ZDB, obviously.
Have you considered offering your assistance to the Chinese in drawing up their next five-year plan?
Is there a link anywhere to his March 17 admission. Otherwise, hard to say what he admitted to.
Jake Haye
That is an extremely good point, and relates to what I was getting at with my second point to Richard T above about dependence on precip (which doesn't relate neatly to global mean temperature). There are of course other important dependencies.
If you read the FUND page in the link in Richard's post, you'll see that he does not oversell it and indeed makes it clear that it should be taken too literally - it's just a tool to help illustrate the relative pros and cons of different policy options, and I'd imagine the numbers that come out are only really meaningful as a way of comparing one set of choices with another and are not intended to be predictions in any way (Richard T, feel free to correct me if I'm misrepresenting you here!)
Don't worry, climate scientists (well, most of us) do *not* think all these things really are a simple function of world temperature - if they were then we'd be out of a job trying to forecast regional changes....
@Jake/Richard B
FUND is calibrated to more elaborate impact models. Where possible, we calibrate to multiple models so as to capture the range of opinions. These models publish their outputs in details, but only sketch their inputs -- often no more than the global mean temperature and the name of a General Circulation Model. So, there are practical limits to the complexity of a meta-model.
We could abandon the meta-model approach and build specific impact models, but then we would lose the broad, encompassing nature of FUND which, I think, is necessary in cost-benefit analysis. If we go down that road, we would quickly be accused, and rightly so, of cherry-picking.
@BigCityLib
I cannot release Ackerman's email without violating his right to privacy. I've asked him for permission.
I'll ask again:
Mr Ackerman: May I make public your email to me dated 9:29 PM 17 March 2011?
Alternatively, someone in the USA could FOI Ackerman or someone in Ireland could AIE me.
I can't keep up with the number of multiple venues for this debate. Further comments from me can be found on Triple Crisis (http://triplecrisis.com) and RealClimateEconomics (http://realclimateeconomics.org/wp). If you want to comment on my work in a place where I'll see it, please use those in the future, rather than Bishop Hill.
To Richard Betts: you raise a number of interesting questions about modeling, I'd be happy to pursue them - perhaps easier in a direct e-mail conversation than via blogs. Our original papers answer some of your questions, and I can answer others.
Regarding the e-mails of March 17, I have no idea what is going on. I wrote two e-mails to Richard Tol on March 17; in both of them, I offered to help him publicize both sides of our debate. He declined. I never heard any request to release these e-mails until the blog posts of the last couple of days. He's got my e-mail address, and never wrote to me asking permission to release these e-mails.
Here's the shorter of my two e-mails to Tol on that day; I have no objection to his release of the longer one; it just says the same thing, and also repeats a number of points made at length in recent posts.
Dear Richard Tol,
As I tried to explain in my e-mail, I believe that we disagree about technical judgments, and those judgments determine whether or not there is a material effect on the results.
I do not think we will reach a resolution of this disagreement through further e-mails reiterating our points of view to each other.
Although you have not persuaded me to change what I wrote, I do think that your disagreement deserves to be heard. If you would like to draft a statement about your disagreement and show it to me, I’ll show you my response before posting or distributing it. I would then be happy to have both of our statements posted on any blog sites where my comments appeared. If we pursue this approach, I will do my best to ensure wide circulation of the two comments.
Regards,
Frank Ackerman
I can't keep up with the number of multiple venues for this debate. Further comments from me can be found on Triple Crisis (http://triplecrisis.com) and RealClimateEconomics (http://realclimateeconomics.org/wp). If you want to comment on my work in a place where I'll see it, please use those in the future, rather than Bishop Hill.
To Richard Betts: you raise a number of interesting questions about modeling, I'd be happy to pursue them - perhaps easier in a direct e-mail conversation than via blogs. Our original papers answer some of your questions, and I can answer others.
Regarding the e-mails of March 17, I have no idea what is going on. I wrote two e-mails to Richard Tol on March 17; in both of them, I offered to help him publicize both sides of our debate. He declined. I never heard any request to release these e-mails until the blog posts of the last couple of days. He's got my e-mail address, and never wrote to me asking permission to release these e-mails.
Here's the shorter of my two e-mails to Tol on that day; I have no objection to his release of the longer one; it just says the same thing, and also repeats a number of points made at length in recent posts.
Dear Richard Tol,
As I tried to explain in my e-mail, I believe that we disagree about technical judgments, and those judgments determine whether or not there is a material effect on the results.
I do not think we will reach a resolution of this disagreement through further e-mails reiterating our points of view to each other.
Although you have not persuaded me to change what I wrote, I do think that your disagreement deserves to be heard. If you would like to draft a statement about your disagreement and show it to me, I’ll show you my response before posting or distributing it. I would then be happy to have both of our statements posted on any blog sites where my comments appeared. If we pursue this approach, I will do my best to ensure wide circulation of the two comments.
Regards,
Frank Ackerman
Mr Ackerman has now agreed to the release of his emails to me. See
http://realclimateeconomics.org/wp/archives/1070
At http://triplecrisis.com/for-whom-the-blog-tols/, Mr Ackerman writes that he "made a one-line correction to the code”. A single change to its code can affect the integrity of a model.
However, in his email of 17 March 2011, Mr Ackerman mentions more extensive changes to the code: "This appears to depend on the method used to remove the singularity. Anthoff’s method was [incorrect description follows]. My methods, described in the paper, involved small changes in the equation or Monte Carlo variable to eliminate the singularity."
That is, Mr Ackerman changed a line in the code and removed a crucial part of the post-processing.
It is good to see Mr Ackerman responding in a reasonable way, here, but I have to say if Mr Tol's description of his actions is correct then Mr Ackerman has previously behaved very poorly indeed. Hopefully the open dialogue can continue.
The "crucial post processing" is presumably the trimmed mean. This trimming does not fix the issue as explained by Nordhaus here: http://frankackerman.com/Tol/Nordhaus_comment_on_Tol.pdf
Anyway great to hear that it has been fixed in newer versions of the FUND model.