Friday
Sep032010
by 
Bishop Hill
Bishop Hill The Climate Code Foundation
Sep 3, 2010 Climate: other This looks interesting:
We are pleased to announce the creation of the Climate Code Foundation. The Foundation is a non-profit organisation founded by David Jones, Nick Barnes, and Philippa Davey to promote public understanding of climate science. The Foundation will continue work on the Clear Climate Code project, and also related activities, encouraging climate scientists to improve and publish their software.
The Foundation intends to work with climate scientists, funding bodies, national and international organisations, and science publishers. We hope to establish climate science in the forefront of science software quality and transparency.
Reader Comments (79)
I'm sure it's a good idea, and like all the other organisations which start with good intents, how long will it be, one wonders, before advocacy and fakery worms it's nasty way in once more?
'Nato chiefs are reportedly concerned about frequent airspace incursions by a new Russian heavy bomber with a distinctive tall, backwards-sloping tail fin. Code-named 'The Hockey Stick Ilyushin.'..........Sorry, only kidding, loving 'the book'. Keep up the good work !
One of the founders is:
Philippa Davey is an administrator for a National Park Authority (Lake District);
Quote, NPA (Engalish National Park Authorities) , "Climate change means that some of England’s best loved landscapes are set to change. From the uplands of the Lake District and Northumberland, to the lowland heath of the New Forest, and the waterways of the Broads – all the National Parks will be affected from increased risks of fire, flooding or erosion."
Goal of CCF, "The Climate Code Foundation goal is to promote public understanding of climate science."
Sort of pre-determined outlook on display here, an acceptance of ideology rather than openess and transparency.
Mac
If she can get climate scientists to publish their code, who cares?
Sorry Bish, but the road to hell is paved with good intentions.
It seems to me the intent is that more openess and transparency will help persuade more people to hold the correct view.
What if the CCF process comes up with an answer that they or climate scientists don't like?
Have they considered that?
Sounds like the same spin as in the last days of a dying govt.
The polices are good its just the way they are being presented thats wrong.
one issue. Quality of the weather station data into the code
I understand the source of your cynicism, Mac. Like all of use, we've heard it said and never seen it done so many times before. But our experience of old does not change the fact that this new pursuit is honourable and needs support.
If the pursuit becomes dishonourable then it will be ripe for the picking and deserving of a kicking. Unless and until then, the stated goal of the project is to support and extend transparency and access to climate data. There is nothing at all, in any way, to criticise in that mission statement. Yes, the proof of the pudding is in the eating. Give it a chance to bake, first.
The main push behind this a company called Ravenbrook
http://www.ravenbrook.com/consultants/
One of the guys submitted to the CCE Review:
CCE Submission (PDF)
One of my areas of expertise is IT and Quality Assurance. After reading their website I have an opinion, but I will let others form their own first.
So I found a link, again people make what they will of it.
Dave Jones is also a Curator at the Comprehensive Knowledge Archive Network, Climate Data seciton. On this page (http://www.ckan.net/group/climatedata) we have,
The climate data group is supported by the Open Knowledge Foundation, Clear Climate Code and the scientists at Real Climate.
So there is a clear Real Climate link. So based on that I would have second thoughts about contributing to these guys due to that link alone. Purely based on the fact that you would not know what information would be used against you.
"We hope to establish climate science in the forefront of science software quality and transparency."
They have thousands of miles to walk and ranges of mountains to climb to put Climate Scientology in the forefront of anything to do with quality and accuracy.
Well done JC.
CCF are linked to RealClimate, and in a blink of an eye the whole edifice crumbles.
Claims of a quest for openess and transparency has been exposed as a lie.
Hell awaits.
Well it is defintiely closely linked to Real Climate...
http://blog.okfn.org/2009/12/05/climate-change-climate-sceptics-and-open-data/
.
.
.
In response to the demand for data, Real Climate (who were also hacked, and who provide two excellent posts on the CRU hack and background context) have published a very useful list of public climate datasets as well as a blog post asking the climate science community for further suggestions.
We’ve been in touch with the folks at Real Climate, and they’ve been supportive of the project and encouraged us to reuse and build on their list of data sources.
So you have work that goes against the "climate scientists" running the "privately PR company funded" website Real Climate. Would you give these people any prior knowledge of your research?
Yes there is a lack of trust. I do not see having close communication channels to Real Climate is going to improve that trust.
Thanks for the link, BH. I remember meeting you at Richard Drake's OCI-1 meeting. I haven't had time to follow up on that; has it gone anywhere?
Regarding detective work done by your commenters: it's really not necessary. My views on climate change are well-known, or at least easy to discover. However, I have always invited and welcomed contributions to these projects from all quarters.
[BH adds: Evening Nick. It's on hold until Richard can get away from his day job. Mid-Sept I believe]
This should be their combined motto and mission statement:
"The raw data. Only the raw data. Everything else is meaningless."
"The Climate Code Foundation goal is to promote public understanding of climate science". There is a world of difference between that goal and creating open and transparent climate software; which is a good goal. The three founders have no expertise in climate science, so how do they know that they are promoting good climate science. It comes back to the Real Climate propaganda of climate science (CAGW) that they will be promoting and will want the public to understand.
Nick B was instrumental in porting the NASA Gistemp code to Python. Many regard this as A GOOD THING.
I think Nick funded this himself through Ravensbrook initially, so I'd imagine that the Foundation is one way to attempt to get some funding for the work into the future.
@Nick Barnes, no implication was given about things being hidden, but...
Perhaps your links to Real Climate should be made clear on the site? The assistance they have given?
Whether you realise it or not, Real Climate is a "brand". Now you chose to work with that brand. Which is fair enough enough. However, to others that "brand" has a significantly lesser value.
And this has nothing to do with data and code. A sourceforge account can be set in 5 mins for open sharing of that. This is about trust. And you have chosen to be linked with an organisation that is not to be trusted.
My opinion means little because I am an IT guy, I will not be contributing code or data. But you used the brand "Real Climate" there are advantages and drawbacks to this.
http://www.zdnet.com/blog/healthcare/alzheimers-and-open-source-medicine/3929
"Open source developed in software as an outgrowth of the Internet, as people realized that collaboration could be done cheaply and could create projects no one company could fund on its own. That same idea has now come to medicine, it’s being proven to work, and I predict it will soon sweep the sciences."
The sooner it sweeps climate science the better, as far as I'm concerned. As a true skeptic, I really want to know what's going on, and if this project can result in any better collaboration between groups and better quality analysis software, surely this can only be a good thing.
@Chuckles
I have been in the IT industry for nearly 30 years. And for a relatively uncool old git, I have been immersed in the Open Source world as well.
A small group of "volunteers" to do this? My interest was piqued. What are the motives? Forgive me but even in the Open Source world altruism can be a thin veneer. Well after following the trail around a bit I can make an educated guess. But it is irrelevant.
The CCF (Ravensbrook) people targeted certain groups in certain ways. They have a clear strategy. The website is scrupulous (too) is making no statements about AGW.
So the question you have to ask what is in it for them? What is their real motivation?
But even the title of the blog post is an implied negative "Sceptics are welcome". "I do not mind if my daughter marries a black man." It is implied superiority/inferiority bias.
Use of Real Climate shows a complete lack of awareness of the universal debate.
Climate science - Real Climate = 0
Actions speak louder than words, people on this side of the debate are not just to be humoured. Some respect needs to be shown.
But this is all just words, the proof will be in the pudding. I have long been thinking that an "open source" model needs to be used here, but a "true open source model", not this.
I've got a few minutes to respond to some of the more far-fetched comments here.
Firstly, my company Ravenbrook paid for us to go to PyConUK 2008 and to OKCon 2010, where we talked about CCC-gistemp and handed out some promotional mugs. They're paying our fares down to Exeter next week as well. Also some of the websites are hosted on a server belonging to Ravenbrook, so the company has implicitly funded that to the tune of maybe a pound. But that's the full extent of the company's contribution. Neither David nor myself has ever been paid a penny for the time we've spent on these projects, which amounts to many thousands of pounds lost earnings each. Now I'm putting my paid work entirely to one side for some months to focus on CCF, until I run out of savings.
Secondly, Real Climate. We have no involvement with them. I like the site, sometimes I comment on threads there, but they have never provided any "assistance" to our projects, beyond a couple of mentions in posts. I've had a beer with WMC, but only after he left Real Climate (and the BAS). I am far more closely linked to Bishop Hill (for instance, I have sat next to BH for two hours at a meeting). The OKFN climate data group, of which David Jones is a curator, is "supported by" Real Climate and CCC in the sense "thinks it's a jolly good idea". No money or other resources are involved.
Thirdly, "Only the raw data". What? Here is some raw data: 31 46 53 21 93 45. Does that tell us anything about climate?
Lastly, maybe someone could let me know how http://ccc-gistemp.googlecode.com/ is not "true open source"? Or do better. Put up or shut up.
In other words, "the proof will be in the pudding". Indeed. The proof is in our pudding and has been for quite some time. Where's your pudding?
They do say they are producing the code under an open source licence, looking quickly, I couldn't see which one BSD, GPL, etc, their own?.
I detect a position starting from a belief in this and an interest in managing an obvious problem, even so, the Open Source world is anarchic, even fractious. Using it as a PR gesture seems dangerous. Then we go to the vexed question of the quality of the data they are writing the code to analyse.
I can't see it's a bad development, because attempts to steer it are going to fall flat.
cosmic: our CCC code is (and always has been) 2-clause BSD. It's right there on the front page of the GoogleCode project. I've been using open-source software every day for more than 20 years, and writing it as a large part of my job for about a decade; I have no intention of "managing" hundreds or thousands of open-source climate science software projects. What we hope to do is to teach climate scientists how to do their own.
And the idea that this is "a PR gesture" is really, really offensive.
I sense that climate science tribalism is about to get the better of this thread, which I think would be a pity. I don't think it's fair to assume that Nick is doing this for anything other than bona fide reasons. You can't assume that because he is convinced by the case for AGW (I think this is a fair representation of your position Nick?) he is somehow untrustworthy. Sceptics have complained vociferously that climate science code is (a) badly written and (b) not public. It seems churlish to criticise someone for starting a drive to do something about this situation, no matter what their views on the science.
Thank you, BH (and yes, you accurately describe my position).
I would like to see them conduct and publish their comments on the code released along with the climategate emails. I recall some saying that it was a bit of a shambles, and the commentary provided by a programmer working on it included expostulations of dismay.
What other 'climate code' is of interest? PCA for temperature reconstructions? Proxy calibrations? GCMs? Temperature adjustment codes and protocols? Potentially important stuff since the output of computer programs have played a major role in climate alarmism over the past 30 to 40 years. It would be good to see more about how, why, or indeed whether, we can trust them.
My point is not that is is a PR gesture, but assuming for the purposes of discussion it were, it would be likely to miscarry, by virtue of being Open Source.
cosmic: agreed. Sorry, maybe I'm a bit touchy today.
Neither Joe Romm, Gavin Schmidt, nor anyone else remotely associated with Hansen or GISS/NASA's shut-eyed cultist site called RealClimate has any slightest credibility. As always, "consider the source"-- complete, transparent coding and raw data, Yes; anything at all besides that rates an emphatic No.
my pudding? A not so subtle dig? My a*se is not that fat, but here is a not so subtle response to that. Whatever my pudding may be, it does not have to feed a consultant company with revenue earned off the back of others hard work and trying to corner the market in a particular expertise but I digress.
BH: This has nothing to do with tribalism, or even an implication of conspiacy. This has got to do with poor judgement. Is it ok to use a brand for your advantage when it suits you. To then turn around and say, "but they are not really that useful" does not work. It makes no difference what my opinion is. The "marketplace will decide". It does show that there was a lack of awareness. Therefore a lack of understanding. If there is a lack of understanding then how can they be an arbiter of the whole spectrum.
Analogies are dangerous things. But here is one...
Code Foundation: Hey Linus (Torvalds creator of Linux) we would like to volunteer to run all the code repositories and provide advice on how to run the projects? We think Linux guys are cool.
Linus: I just checked, didn't you tell everyone you had lunch with Bill Gates last week?
Code Foundation: Yes but we only talked about sailing in Seattle harbour. And besides I also sat next to Richard Stallman (true open source advocate) once at a conference.
Linus. No thanks.
The blog entry should perhaps not should have been "We Welcome sceptics" but "We welcome all who think open code and data..."
Rather than mentioning Real Climate when it suited them, perhaps a real DISPLAYED knowledge of the climate debate universe would have better served their stated aims.
I wish you well with your consultancy company, I see you already have your foot in the door with the Met Office.
I have nothing more to add, this is not about tribalism, it is about poor judgement.
Let the "marketplace" decide.
Jiminy Cricket is trying to be very rude about my motives and judgement, but he completely fails because his comment really doesn't make any sense at all; it totally fails to connect with reality.
"feed a consultant company"? CCC has never seen a penny change hands and has cost my company large sums in opportunity cost.
"others hard work"? No, our own - check the submission logs.
"use a brand for your advantage"? Meaning Real Climate, which brand we have never used for our advantage or otherwise.
"arbiter"? Um, no.
He is plainly living in some bizarro fantasy world. Given the tone of this thread, I'm not going to comment here again. Thanks for the link, BH, and for the positive words.
As a grumpy old git whose training was founded upon slide rules, log tables and brunsvigas, can somebody point out where I am going wrong here?
Many years ago, a group of blokes in the US were receiving log sheets from all over the world from human observers of their local temperatures and they fiddled about with the figures and finished up with a giant spreadsheet.
They decided that it might be interesting to process this information so that they could finish up with a graph of temperature against time.
It was probably realised that if they used slide rules and brunsvigas they would have all died before they got anything worth while, so they got a new fangled computer and bought a copy of Fortran and lo and behold in no time at all they had a graph.
Fast forward a few decades and along comes another group of blokes who have a new bit of software which is considered superior to Fortran. They get together with the Fortran guys to find out what they were doing with the information in the database and they then write new routines in the new software to do exactly the same thing on exactly the same database.
Blimey, they get the exact same graph, as you would expect, surely. So either old Fortran or new bells and whistles gets the same graph. That is probably a good thing because the Fortran guys can now use up-to-date software and get the same graph and thereby avoid being called old fashioned.
However, I was under the impression that although the use of Fortran was sneered at by the younger elements, it was the database itself that was regarded as a bit iffy in that the elements of the data were under continuous massaging and tweaking. This happened whenever someone caught them out making claims that were not supported by the database as it was at the time of the catching out. So, to avoid having to admit the mistake, the database was fixed to reflect the original claim.
So what has happened here? It seems to me that whilst getting the same answer using up-to-date software may go down well in certain circles it does nothing to resolve the overarching problem of the database. Surely this is what should be being looked at, not farting around with with calculating machines.
Regardless the views of those spearheading the project, Converting GISTEMP to Python from Fortran is a noble endeavour, indeed. For those interested in determining if GISTEMP actually does what it says it does, the auditing task will no doubt be significantly simplified.
I have a question for Nick B. I remember a while back that it was mentioned on your site that you had discussions with Reto Ruedy from GISS about the possibility of NASA actually adopting the CCC version of GISTEMP. Did that ever go anywhere? And if not, is there going to be some process in place for forward-fitting changes to GISTEMP code made by NASA (which I assume will inevitably happen) into the CCC version so that the two code bases are logically in sync? How will that work?
Thanks,
Dave
To Brownedoff:
Though I've never looked at the GISTEMP code (and I'm not an expert on this subject), I believe the issue is that GISTEMP is the program that is manipulating the data to produce the final temperature product published by GISS. Manipulations include UHI adjustments, adjustments to homogenize the data, etc.
There are two main questions with respect to these manipulations: a) Are the manipulations justified?; and b) Are the adjustments (like them or not) implemented correctly in GISTEMP?
I think the CCC project (and please correct me, someone, if I'm wrong) is attempting to address question b). By converting the GISTEMP code to Python which is much more readable than Fortran, and by getting rid of ancient software engineering techniques which were likely necessitated by computing limitations that don't exist any longer, it will now be a lot easier for someone to verify that GISTEMP is correct with respect to its stated goals. Everyone benefits from this.
As for question a), if you don't agree with the manipulations then I guess you better prosecute this case in the science journals.
Dave
Toad: The Hockey Stick Ilyushin must necessarily have a NATO designation begins with "b" > Bishop
Ferguson. You are right of course. 'Bishop' will do nicely . On the other hand a squadron of nuclear armed 'Bishops' crossing the North Sea has a distictly ominous ring !
Quote, Nick Barnes, "Well, I'm quite enjoying Pearce's book: it seems approximately "fair and balanced" (not in the Fox News sense). Certainly a lot, lot better than either the Montford or the Mosher+Fuller books. As usual the book reviews I've seen online don't seem to be about the same book.
I think it's a mistake to criticise Pearce too harshly. He has been following climate science for a long time, he's capable of understanding it, and he's generally not taken in by septic arguments. He does sometimes write nonsense, and he did really cock up the early call on the CRU emails. But he's much closer to being "a friend of the truth" than most other journalists who write on the area. And the truth needs all the friends it can get."
Bish, Nick Barnes is not a fan.
Note the use of the word "septic", now commonly used by alarmists as a corrupted and perogative term for sceptics.
Nick Barnes protestations of innocence and his intentions being mis-judged are simply hog-wash.
webofbelief Sep 3, 2010 at 8:54 PM
Thank you for confirming my proposition that "getting the same answer using up-to-date software may go down well in certain circles".
Apparently in those certain circles "it will now be a lot easier for someone to verify that GISTEMP is correct with respect to its stated goals. Everyone benefits from this."
So, in the past whilst it was possible, albeit with difficulty, to verify that the Fortran results were OK, all this new work does is just to make it easier! How very up-to-date. I think that the use of the word "everyone" is over-reaching - you probably mean a handful of blokes in Exeter.
I think your final paragraph is irrelevant.
All I said was that "I was under the impression that although the use of Fortran was sneered at by the younger elements, it was the database itself that was regarded as a bit iffy in that the elements of the data were under continuous massaging and tweaking. This happened whenever someone caught them out making claims that were not supported by the database as it was at the time of the catching out. So, to avoid having to admit the mistake, the database was fixed to reflect the original claim".
If my impression is correct, then this sort of fixing is not "manipulation" in the sense that you describe it (as a software process) in your first paragraph.
Correection: Note the use of the word "septic", now commonly used by alarmists as a corrupted and pejorative term for sceptics
"The Foundation intends to work with climate scientists, funding bodies, national and international organisations, and science publishers"
\immediately falls asleep
BTW, I hear the new swine flu vaccines are giving people narcolepsy.
(Dear Bish, we do understand your positive take and attitude, and do not wish to beat it down. I think we (or atleast I) are just trying to look at the other side - you cannot cordon off, and organize curiosity and 'out-of-the-box' thinking (or whatever fancy you want to call it))
Sorry Bish but Nick Barnes is a true believer. The CCF is all smoke and mirrors.
"if you don't agree with the manipulations then I guess you better prosecute this case in the science journals"
This appeal to the authority of "science journals" is an obvious indicator that the person appealing recognizes the validity of criticisms of the manipulations, but lacks the fortitude to exercise their own judgement.
Is this OK mom?
Andrew
I think some people are being a bit cynical about motivation. When Steve McIntyre was over, he made a comment that one reason CRU may have had for not releasing code was because if they had, people would realise just how banal much of CRU's day job was. And potentially how ill equipped they are to peform the tasks they've been paid millions to do. Harry's readme seemed pretty clear on that point.
For most people, probably including climate scientists the computer stuff is probably boring. They're not professional IT types and wouldn't know a formal method if it snuck up behind them and hit them over the head with a hockey stick. This has been a theme throughout the climate debate. Bad coding, bad maths. GISS got turned from a clunky cobbled together bit of fortran into something more modern just because people could. And people did it for free because it interested them.
It's much the same issue as business has. I've seen many businesses that have allowed home-grown projects get embedded into that business. Trying to formalise it is often very painful, but often necessary for boring stuff like maintenance and SOX compliance. They'll still ask IT for a copy of Visual Studio, but can't articulate a requirements specification so people can do what they want properly.
Climate science has kind of articulated the requirements, even if the data is often lacking. If the project remains fully open, then it should be easy for 'sceptics' to see if code or data is being misapplied. I think it's a good thing, but may not be such a good thing for institutions who rely on grant funding to hack code together.
To Brownedoff:
Yes, I guess I mostly agree with you (I think). I was just trying to summarize the two issues involved, perhaps for my own benefit as much as anyone else's and I think I made a few poor choices in my words.
For one, I think the word "manipulation" wasn't very clear (I agree). I was just trying to say that there exists some algorithm that is applied to the raw data to produce the final temperature product. This algorithm could be performed on a computer or manually, it doesn't matter. The appropriateness of the algorithm (i.e., does it make sense?) is something that should be settled among the interested parties in an appropriate forum. And I agree that the CCC project doesn't really address this issue, so far as I can tell.
However, I do still think there is value in producing a new code base for this piece of software (for the CRU one as well). Not only will it be easier for us to verify that the current algorithm is implemented correctly (I don't know why you think this is not important), but it will also be much easier to maintain this software in the future and will help ensure that the software remains correct with respect to the published algorithm, which will likely continually change over time, for better or worse.
To Bad Andrew:
I'm not a big fan of appeals to authority, so if that's what I did in my last post, then I would like to retract it.
As I said above, I simply meant that the issue of the algorithm's appropriateness is something to be dealt with in an appropriate forum and is a separate issue from the correctness of its implementation in the GISTEMP software. I think feedback from the blogosphere is perfectly fine and hopefully that will lead to refinements to the algorithm that are accepted by mainstream climate scientists.
- Dave
Dave,
"accepted by mainstream climate scientists"
Whoops... you did it again. ;)
Andrew
Consensus, appeals to authority, the opinion of the mainstream? No, thanks.
Your gracefullness should not be ever so ready to accomodate.
webofbelief Sep 4, 2010 at 12:15 AM
Look, we all now know that you are vey very pleased that you, and others in that certain circle, can now check the output of the data mincing machine with less effort that was previously required. Good. Lets agree to to drop this aspect of the exchange and please stop trying to deduce what I think. However, do not hold your breath awaiting the implemenation of this data processing utopia in the shady world of temperature manipulation.
I will come back to my original point: all that these CCF people have achieved is to provide an opportunity for the butcher to move from using a cast iron mincing machine in a gloomy candle lit dungeon to a well lit area working with a glass mincer. The mince is the same whether it is from the cast iron mincer or from glass mincer. Also, it remains to be seen whether the butcher takes up this opportunity.
However, the butcher will still be working in a meat preparation area to which access is denied to all except for the favoured few, and they are desperate to keep it that way. It is thought that, in the case of a few temperature reporting stations in Oz and Nz, the butchers have been chucking cubes of cheaper cuts into the mincer and yet continue to pass the resulting product off as top quality steak mince.
The CCF blurb says "The Foundation is a non-profit organisation founded by David Jones, Nick Barnes, and Philippa Davey to promote public understanding of climate science." If this is what they believe they are doing, then they will have to do much more than merely provide an up-to-date date data mincing machine. We need to know what the butchers are up to in the back room and if CCF are unable or unwilling to get stuck into this problem then they need to edit their mission statement accordingly.
It is not a question of moving this back room problem to another forum, because CCF opened the door for comment on it by publishing their overblown and misleading prospectus.
Brownedoff Sep 4, 2010 at 5:40 AM
The description of the adjustments that are made to the data by GISTEMP are documented and available on the GISS website (see below), and GISS say the document is updated every month.
http://pubs.giss.nasa.gov/docs/2001/2001_Hansen_etal.pdf
So the following are all available for public scrutiny:
1) The GISTEMP adjustment algorithm.
2) The GHCN data that is used as input to the GISTEMP program (I believe GISTEMP uses the unadjusted GHCN data; there is an adjusted one as well). Just go to the NCDC website to get this data.
3) The GISTEMP code itself, in Fortran and also now in Python.
You say "the butcher will still be working in a meat preparation area to which access is denied except for the favored few", but which part of the "meat preparation" process are you specifically referring to? The analogy is just getting in the way at this point.
I understand that there are lots of legitimate concerns out there about the way GISTEMP adjusts the data. I'm not denying any of that. But perhaps you can tell us more specifically which of the documented adjustments you don't agree with.
- Dave