Buy

Books
Click images for more details

Twitter
Support

 

Recent comments
Recent posts
Currently discussing
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« The forbidden history of unpopular people | Main | Science as a public enterprise »
Wednesday
Jun062012

A dangerous practice

Ed Hawkins points us to this paper, reporting a seminar on uncertainty. Specific issues considered are choice of statistical methods and openness in that regard. One possibility that was discussed was whether codes should be made available:

One obvious way to facilitate auditability and reproducibility is with provision of codes which operate on the raw measurement inputs to produce data products. Even in this relatively small subset of participants there was a wide range of perspectives regarding code provision. Some are willing to provide code, although there is doubt whether code alone will explain methodologies and whether external users can simply copy code and reproduce results. Although code provision can directly explain what and how analysis was done, it may be silent on why the particular methodology and implementation was selected. Some agencies have extremely strict requirements for code release, insisting on near commercial-grade quality, which most researchers do not have the time or resources to develop. Therefore, code release is not a feasible option under these conditions. Another subset of statisticians feel that it is better to provide inputs and outputs to interested parties. Supplying (non-peer-reviewed) code can be a dangerous practice, while leaving it to the user to develop the code and reproduce results independently encourages thought and engagement by the potential user. In this replication of results, real scientific value is realized through the analysis of structural uncertainty.

Oh well.

PrintView Printer Friendly Version

Reader Comments (40)

If code isn't well enough structured to even let other people see it, how confident can researchers be that it's correct?

Jun 6, 2012 at 1:43 PM | Unregistered CommenterNial

Releasing code is indeed dangerous -- see my recent interchange with Frank Ackerman who took our code, introduced a bug, and told all the world how stupid we are -- but not as dangerous as not releasing code.

The way forward lies in better code and better protection.

Jun 6, 2012 at 1:43 PM | Unregistered CommenterRichard Tol

Although code provision can directly explain what and how analysis was done, it may be silent on why the particular methodology and implementation was selected.

The paper is supposed to explain why a particular methodology and implementation was selected.

Some agencies have extremely strict requirements for code release, insisting on near commercial-grade quality, which most researchers do not have the time or resources to develop.

Perhaps some examples of this? It is more than likely that they insist on commercial-grade quality when code is released for applications and not as supplementary materials for a research paper.
Another subset of statisticians feel that it is better to provide inputs and outputs to interested parties. Supplying (non-peer-reviewed) code can be a dangerous practice, while leaving it to the user to develop the code and reproduce results independently encourages thought and engagement by the potential user.

Supplying inputs and outputs should already be a requirement. The problem with only supplying a description of the method is that plain language is so ambiguous it is very nearly impossible to accurately describe a complex process.

Jun 6, 2012 at 1:50 PM | Unregistered CommenterTerryS

Richard

Looks to me like Frank Ackerman showed the world how stupid he was/is.
Bad experience but great response.

Jun 6, 2012 at 1:55 PM | Unregistered Commenterpesadia

I think this is a complex situation. From my personal experience I have learnt more when trying to replicate results with separate code first, whilst discussing the methodologies chosen, before later swapping code to understand any remaining differences. Of course, this requires openness and engagement from all. For example, the recent efforts by "Clear Climate Code" to rewrite the GISTEMP code in Python worked well I think.
Also, that there is such a wide set of views from professional statisticians is especially interesting.
Ed.

Jun 6, 2012 at 2:04 PM | Unregistered CommenterEd Hawkins

Speakers at the meeting that led to this paper included Menne and Williams, two of the authors of the error-filled and unreleased GHCN adjustment code.

Menne's talk at the above link is interesting. It shows how the adjustments create warming in the US, see slides 24 - 31.

Jun 6, 2012 at 2:11 PM | Registered CommenterPaul Matthews

Translation: If we release the code people will find out it didn't work and we fudged the results.

Jun 6, 2012 at 2:31 PM | Unregistered CommenterStuck-record

@Pesadia
The world did not need further proof of the venality of Frank Ackerman.

Code is dangerous in dishonest hands. In this case, we lost an inordinate amount of time. We also lost a commercial project.

We now have our code watermarked so that tampering is readily detected, but as journal editors and referees do not demand to see watermarks, anyone can pull an Ackerman.

Jun 6, 2012 at 2:53 PM | Unregistered CommenterRichard Tol

It seems that as with statistics, so with software development - never consult the experts because "we" know best.

It's a simple matter to put code into a version control system like Subversion and then publish that repo on the web as the definitive source for your software release. Of course, if you use a VCS from the start, those with an interest can also see the history of the code, including who changed what and when - so no reason to worry about provenance of defects, why changes were made and by whom etc.

Publishing code in this manner is the best form of peer review out there. It's how open source projects work - find a defect, look at the source, fix, check back in (either yourself or through a project team member).

Of course, ideally all of this code should be supported by unit and integration tests. These days, most of us in the software world consider well-written tests to be a statement of intent for the production code - in other words, the definitive spec for the work.

Jun 6, 2012 at 2:58 PM | Registered Commenterthrog

This is just typical nonsense from academics who have never developed any sizable amount of code.

If published papers or important decisions are being made on the basis of output from software code then that code must be able to reproduce these results (or at least a known version of it) otherwise the results cannot be accepted,

Furthermore, if the code is being written and maintained with the quality control expected in today's world, then there should be substantial comments within it to explain the thinking behind the code. If it does not have such comments then the code and the results should never be trusted.

Excuses like researchers do not have enough time are just covering up poor software and scientific quality.

Jun 6, 2012 at 3:06 PM | Unregistered CommenterConfusedPhoton

Climate science needs competent physicists, not statisticians. Statistics must never be allowed to drive the physics; it is like having your dog invest your money by how much it barks at a given choice--"look, he likes it, that shows the science is settled!"--there is no sense to it, it is not good science. And climate scientists have the physics of atmospheric warming literally upside down (it is not warmed from the surface) and causally reversed (temperature controls atmospheric carbon dioxide, not vice versa, and surface temperature--which is an effect of the thermodynamic processes-- is being used as an unrecognized cause, i.e. as radiative forcing, via the fundamental, false assumption of "blackbody surface emission and backradiation"). Statistics--when it is not allowed to fail (they simply ignore the dependence of atmospheric carbon dioxide on temperature, and swear they are measuring blackbody emission from the surface), and that failure believed and learned from--simply cannot correct for such unquestioned physical nonsense. The debate is insane, talking about it is misdirected--just cancel the tyrannous climate policies, and throw the science bums out with their unworkable models.

Jun 6, 2012 at 3:16 PM | Unregistered CommenterHarry Dale Huffman

"Some agencies have extremely strict requirements for code release, insisting on near commercial-grade quality, which most researchers do not have the time or resources to develop. Therefore, code release is not a feasible option under these conditions."


Rigorous testing, version control, and documentation of code cannot be considered optional or insignificant, at least in any area of public importance. If quality analysis is not considered part of the remit of individual scientists and labs, then the "field" as a whole needs to come up with a way to test and certify code used.

Claiming that it's just too much to do or too expensive etc. cuts both ways.... it points to the weakness of claims that vaunted "peer review" of journal submissions can possibly be adequate. It seems that all that a peer reviewer can certify is that an article "seems" loosely credible to him/her (the reviewer), not that any rigorous analysis has been performed independently of the authors.

Jun 6, 2012 at 3:18 PM | Registered CommenterSkiphil


Furthermore, if the code is being written and maintained with the quality control expected in today's world, then there should be substantial comments within it to explain the thinking behind the code.

Actually, that's a bit old school these days. ;-)

Well-written code shouldn't need much in the way of comments. To a large extent, it should be self-documenting, using well-chosen names for methods / functions (using verbs, e.g. "makeCupOfTea") and attributes / parameters / variables (using nouns, e.g. "spoonsOfSugar").

This is preferable because as the code base matures, some developers will inevitably update the code without updating the comments, so they drift out of line. It shouldn't happen, but it does, especially if the project manager is pushing changes through in a rush.

A classic interview question is to ask a developer to explain what some code does ... with the comments out of line with the actual code.

Jun 6, 2012 at 3:23 PM | Registered Commenterthrog


Climate science needs competent physicists, not statisticians.

In a hybrid discipline of such seeming importance, surely they need competent physicists, statisticians *and* coders? ;-)

Jun 6, 2012 at 3:25 PM | Registered Commenterthrog

As a matter of principle any code produced at public expense to generate results in a peer reviewed paper should be put in the public domain. When I work with private sector clients, the contract often includes a clause requiring me to give up any intellectual rights to software produced during that contract. Similar rules should apply to government funded research.

Jun 6, 2012 at 3:55 PM | Unregistered CommenterRon

throg. I think everyone of us in software development has seen the value of this almost instantaneous peer review of ones work.


"while leaving it to the user to develop the code and reproduce results independently encourages thought and engagement by the potential user. In this replication of results, real scientific value is realized through the analysis of structural uncertainty."

Is it just my reading or are they trying to make a virtue out of hiding their code?

Jun 6, 2012 at 4:09 PM | Unregistered Commenterconfused

If not letting the code be seen is good, maybe not even the developer being able to see their own code is best!

Jun 6, 2012 at 4:26 PM | Unregistered CommenterAC1

Hilarious post on RealClimate.

Are the new numbers realistic? I and many colleagues I spoke to have serious doubts. It is a model result which is in stark contradiction to data-based estimates. The simulation is based on a simple assumption: first the total water demand was estimated, second the availability of near-surface water, and then the shortfall was assumed to be completely supplied by unlimited use of fossil water.

Jun 6, 2012 at 4:30 PM | Unregistered CommenterMikeN

Well, on one hand I did wonder if what I was posting was blindingly obvious, but sometimes even within the software development community I see signs that some of this stuff isn't taken on board.

I suspect that a lot of our academic institutions labour under the same illusion as many businesses, namely that anyone with half a brain can sit down and write "code" as well as a trained, experienced developer.

Unfortunately, like most engineering disciplines, experience and craftsmanship both play a part in writing good quality code.

Of course, as an experienced professional, you would expect me to say that, but I presume there's a reason why my skills are still in such demand. ;-)

Jun 6, 2012 at 4:51 PM | Registered Commenterthrog

Your posts are spot on throg. In my days of systems development the auditors would have failed us as an enterprise if we didn't have competent version control and audit trails. It's most certainly the case that any major bank would not get away without being audited, yet these 'scientists/advocates' are dealing with systems that will spend many,many times the worth of what the banks hold and control.

Jun 6, 2012 at 5:16 PM | Registered CommenterHarry Passfield

Arrrrggggggggggggggh. I'm a Software Developer and nobody, NOBODY uses the phrase "codes" for "code". Code is already plural. Software, programs, tools, code, whatever, but "codes" is annoyingly wrong.

Jun 6, 2012 at 5:42 PM | Unregistered CommenterRobinson

these 'scientists/advocates' are dealing with systems that will spend many,many times the worth of what the banks hold and control.
Not my problem, squire. I just crunch the numbers. What you do with them's your worry. Ain't you read my disclaimer?

Jun 6, 2012 at 5:45 PM | Registered CommenterMike Jackson

It is instructive, in this regard, to observe Jean S & Steve McIntyre attempt to replicate one of the calculations of Gergis et al. on Climate Audit, here. They have [some of] the data (archived by the authors), and a description of the algorithm in words. But they haven't come up with similar results.
.
This could easily be avoided by providing code, whether pretty or ugly. I appreciate the sentiment for developing code independently; I have found it useful to do so professionally. Unthinking reliance on turnkey code is not to be encouraged. But a verbal description of an algorithm will invariably be less precise than code. Publishing the code (or at a minimum, making it available upon request) would seem to be a requirement for replication.

Jun 6, 2012 at 6:38 PM | Registered CommenterHaroldW

"Some are willing to provide code, although there is doubt whether code alone will explain methodologies and whether external users can simply copy code and reproduce results."
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
There is a World Meteorological Organisation pdf concerning metadata and homogenization.
http://www.wmo.int/pages/prog/wcp/wcdmp/wcdmp_series/documents/WCDMP-53.pdf

In some ways the data, metadata and homogenisation processes pose similar problems to the use of code for climate forecasting, statistical evaluation, visualization of data amongst many other things.

The explanation of the rationale for the WMO homogenisation guidelines states:

"Metadata should reflect how, where, when and by whom information was collected. Ideally, a complete metadata should register all the changes a station has undergone during its lifetime, composing what is called the station history."

"Stations and NMHS should guarantee that the algorithms or formulae used to correct and convert elements are adequately documented."

"Again, a minimum required practice would be to report if any homogenization technique has been applied or not."
http://www.wmo.int/pages/prog/wcp/wcdmp/wcdmp_series/documents/WCDMP-53.pdf

Comparable explanations of computer code could be provided with the code when submitted for scientific verification and validation of the results produced by the code. Answering such questions as the reasons a code was used and whether any data was discarded and why, would help to clarify the code and its results.

The WMO have a 'Registry of Software Available for Exchange'. It could be a template for how code, together with its justification and methodologies may be stored so that they are available for independent testing.

"The free exchange of meteorological application software amongst WMO Members helps the distribution, implementation and use of software packages at WWW centres. The CBS Software Registry provides information to Members on the software packages offered by individual Members through the WMO web server."
http://www.wmo.int/pages/prog/www/cbs-software-exchange/introduction.html

Jun 6, 2012 at 6:58 PM | Unregistered Commentermfo

A smokescreen - either you have confidence in your observations and results or you do not.

Even messy code must do the advertised numeric manipulation - and sensibly one must explain what's been done and why.

Fudged figures which converge or trend to a result that is favored by the researchers on a confirmation bias track stinks and we see more than enough of that pass through this blog.

"No result" isn't comfortable for many folk for a variety of reasons - but it's the honest thing to do and it's high time some research folk both volunteered that there wasn't a clear answer and were applauded for doing so.

Human nature being what it is - rigor goes out the window when the budget (or a belief system) is at stake.

Most academic research and it's archiving is publicly funded - it's bad enough outside academia wading though Springer, Elsevier et al (and academic libraries) trying to monetise research papers without the producers being able to hide how they arrived at their results.... The whole edifice could just collapse...

Jun 6, 2012 at 7:27 PM | Registered Commentertomo

Look, we can't publish the Code - you'll just try to show how flawed it is!

Jun 6, 2012 at 7:30 PM | Unregistered CommenterIan E

There are some very distinct issues in this. First, if there is a real phenomenon that is affecting an instrument reading, that effect should survive any appropriate analysis of the instrument record. Second, the raw, untransformed, instrument record is a critical element and is more important to replication than any code. Third, the methodology behind the decisions on how to transform and analyze the data should be absolutely explicit. Fourth, the code as developed and guided by the methodological assumptions needs to be available to determine if it does indeed conform to the methodological requirements. Absent any of these and you are facing serious limitations in substantiating an argument or in supporting replication of your results.

Jun 6, 2012 at 7:38 PM | Unregistered CommenterDuster

Jun 6, 2012 at 7:27 PM | tomo

Most academic research and it's archiving is publicly funded - it's bad enough outside academia wading though Springer, Elsevier et al (and academic libraries) trying to monetise research papers without the producers being able to hide how they arrived at their results.... The whole edifice could just collapse... [emphasis added -hro]

In the Alice-in-wonderland-world of climate science, as articulated by no less a luminary than Phil Jones, "Yuck" is apparently an acceptable expression of scientific concern, while “Intuition” and Because! I! Said! So! will trump due diligence, every time – otherwise the whole peer review process would go “down the tubes”. [See Phil Jones keeps peer-review process humming … by using “intuition”]

This may seem like a somewhat simplistic analogy ... but can you imagine how many celebrity chefs would stay on the airwaves (or sell as many cookbooks) if they were to adopt the approach of "climate scientists" towards replication, transparency and disclosure: "I use eggs, sugar, flour, mix it with this stuff you can buy here and cook it. It's delicious because my pals tasted it and said so." They'd be off the air and out of the bookstores, in no time flat!

Jun 6, 2012 at 8:47 PM | Registered CommenterHilary Ostrov

Harry Dale Huffman: 3.16 PM: 'Climate science needs competent physicists, not statisticians. Statistics must never be allowed to drive the physics'

Agreed. The problem is the 1906 physics IPCC co-founder Houghton used in his treatise on the Physics of Atmospheres. It breaks down at the bottom and the top of the atmosphere. This is because it's based purely on radiation physics when in the lower atmosphere convection dominates and there's different IR physics than assumed.

The latter means there is probably no 240 W/m^ DOWN at TOA, And at BOA, they assume the radiative heat transfer equilibrium between the air and the surface is that you'd get in a vacuum.

Engineers know it's coupled conduction. convection and radiation, with the latter much smaller than the sum of the former. The bottom line is that the energy delivered by the Sun to the Earth's surface and lower atmosphere is multiplied by ~1.4, the IR part multiplied by ~5. This is the origin of the artificial extra warming and the transition to radiation dominance in the models.

The World is now cooling from natural causes despite CO2 increasing. Net CO2-AGW may be kept zero by natural processes.

Jun 6, 2012 at 8:53 PM | Unregistered Commenterspartacusisfree

"One obvious way to facilitate auditability and reproducibility is with provision of codes which operate on the raw measurement inputs to produce data products".

"One way"? This struck me as very curious. For a start programmers don't refer to "codes" in the plural. Code = software. Code = the program. Code is not the binary or hexadecimal representation of the program any more; these days it means the clear (programming) language expression of the required transformation of raw data into "product".

Secondly the relationship between these "data products" and the inputs from which they are derived is the "code", only the "code" and nothing but the "code". The relationship between the output data products and the input raw measurement data is only expressible as the "code" in this sense. That's what the code does; it specifies the function f where output = f (input) or "data product" = f("raw measurement"). The program specifies the relationship directly. If we're not given the program we can't understand what the relationship is or is intended to be. This should be blindingly obvious. You have to declare the "code", the relationship between input and output, otherwise nobody can possibly know what you're talking about.

Why on earth should this need saying?

Jun 6, 2012 at 9:17 PM | Unregistered Commentersimon abingdon

Throg, Snotrocket, Confused

Agree with all your comments. These guys are like 12 year olds learning to generate code, out of their depth at this level. There is no shame in not being skilled in code generation, the problem lies in not admitting it to yourself and seeking assistance. The days of doing "Garden Shed" development are long gone, particularly in Climate Research

Sandy Sinclair

Jun 6, 2012 at 9:23 PM | Unregistered CommenterSandyS

A research student of mine once decided to adopt some code published in a well respected advanced text book. He did, of course, scrutinise it carefully and found two blunders. He wrote politely to the author and his reward was a bravura display of ingratitude, bluster, offensiveness and threats launched from Berkeley (or maybe it was Stanford - I don't have the book to hand).

And it wasn't even Climate Science we were pursuing - just an uncontentious area of thermodynamics.

Jun 7, 2012 at 12:32 AM | Unregistered Commenterdearieme

[Snip - manners]

Jun 7, 2012 at 12:43 AM | Unregistered Commenterbigcitylib

Don't any of you programmers write test code?
I have switched to test driven development (TTD) where you write a test first before any of the other code. Modern development software like Microsoft's Visual Studio makes this ridiculously easy. Any introduced bug should be easily detected. Tests are also an excellent way of documenting the code.
I've been computer programming since the early 1980s by the way.

Jun 7, 2012 at 12:57 AM | Unregistered CommenterGary Mount

Perhaps the ease of writing the code in R or Matlab leads to overly sloppy code -- get them to write their models in Haskell and you'd soon see robust, high-quality code. (Not to be attempted by climate 'scientists' who cannot plot trends in Excel.)

Jun 7, 2012 at 3:21 AM | Unregistered CommenterRick Bradford

"♫It don't mean a thing if it ain't got that swing♫." Swing, in this case, means replicability. Science is not consensus; it's experiment, publication, and replication. If publication doesn't incorporate all necessary methods for replication (including code, codes, or codpieces), it's not Science. It's pretense.

Skiphil is dead-on about peer review being oversold by climatologers.

Jun 7, 2012 at 4:16 AM | Unregistered Commenterjorgekafkazar

The Met Office state that their models are now approaching a million lines of code (probably a lot of this is open source subroutines) so releasing it would probably not tell us much as it would be near impossible to deconstruct without documentation.

I would settle for a clear description of the methodology, with logic flow daiagrams. From what I have read I have been horrified by some statements in the publisged literature, for example that numerical models used for multi-decadal forecasts:

A. Use low pass filters "chosen to preserve only the decadal and longer components of the variability", and,

B. Accomodate errors that cause instabilities "by making periodic corrections, rather
than by fixing the underlying routines"

It is blindingly obvious to anybody with any knowledge of numerical modelling that such techniques completely invalidate the methodology, and render the models unifit for purpose.

Jun 7, 2012 at 11:05 AM | Unregistered CommenterRoger Longstaff

Hilary Ostrov @8:47 PM

Well put - I was struggling for a metaphor - thanks!

Having said that - in my experience I'd say some celebrity chefs have their books ghost written by BBC science journalists...

As an aside - the cost of becoming an external university library member is a source of some small irritation to me - my local ivory tower wanted £250 p.a. last time I asked. Perhaps that's the way it will go - after all - time is money.

I agree with the commenters about VCS use and code documentation - if the code is to be re-used then navigating around it is a key feature. I really think the code (and data) used to produce a publicly funded research result should be freely available - whether you understand it or not - otherwise we're into " because I/we say so" territory - which is not a good move in my view.

Jun 7, 2012 at 11:20 AM | Unregistered CommenterTomO

TomO

Perhaps O/T. Before you by an external ticket, beware of restrictions. I'm having an on-going argument with the local university. As an ex-member of staff, with a life ticket for the library when I retired (£60.00 ten years ago) I now find I am not allowed access to any online journal, as the university, and presumably all the others in the UK, have signed an agreement to say that only current staff and students may have access. I can no longer read the latest journals, as most of them are now only available online and I am effectively blocked from doing any more meaningful research.

Jun 7, 2012 at 12:38 PM | Unregistered CommenterMessenger

Messenger@ 12:38 PM

I recall something about this being mentioned - I gave up after three gos ... as the library staff and procedures changed - the new crew were, it seemed to me, a little obstructive and I got more than a whiff of "what do you want to do that for?" Gatekeepers to the ivory tower did cross my mind.....

Jun 7, 2012 at 5:03 PM | Unregistered CommenterTomO

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>