Bishop Hill Climatologists raise the shutters again
Jun 4, 2013 Climate: CRU Climate: MetOffice Climate: Surface FOI About a year or so ago, the Met Office and CRU announced the release of the CRUTEM4 temperatures series. As John Graham-Cumming noted at the time, they were positively gung-ho in their enthusiasm for transparency:
Given the importance of the CRUTEM land temperature analysis for monitoring climate change (e.g. Trenberth et al. 2007), our preference is that the underlying station data, and software to produce the gridded data, be made openly available. This will enhance transparency, and also allow more rapid identification of possible errors or improvements that might be necessary (see e.g. the earlier discussion of homogeneity adjustments in the SH).
The data was made available at the time, but, according to the Met Office, the code would follow soon after:
The code required to produce the CRUTEM4 fields and timeseries will soon be made available. Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series due to changes in processing methodology.
That's what they said at the time. However, if you look at the equivalent page now, it reads as follows:
Note that code previously released for CRUTEM3 cannot be used to exactly reproduce CRUTEM4 temperature series from these files due to changes in the processing methodology used.
I guess this means that the code is no longer going to be released.
According to JGC, the change to the web page took place after May 17th. I wonder if there is any connection to the Keenan article?
Reader Comments (28)
I posted this earlier. Probably relevant here.
This code in R fetches Met Office data from this page.
base<-"http://www.metoffice.gov.uk/hadobs/indicators/";library(installr);
url<-"http://www.metoffice.gov.uk/hadobs/indicators/11keyindicators.html"
download.file(url, destfile="metoffice.html", quiet = FALSE)
tree<-htmlTreeParse("metoffice.html", error=function(...){}, useInternalNodes=TRUE, encoding="UTR-8", trim=TRUE)
links<-xpathSApply(tree, "//a/@href")
d<-paste(base, links, sep="");d<-d[3:79]
dwnldata<-function(x){
name<-file.name.from.url(x)
download.file(x, destfile=name, quiet = TRUE)
}
lapply(d, dwnldata)
Could be that there are issues with the latest update CRUTEM.4.2.0.0. code/methodology? CRUTEM4 monthly numbers taking far longer to produce than previous issues. Also at least one obviously spurious result has been posted. Has since been revised/corrected/adjusted after being brought to the attention of RB on a BH discussion board.
Following subsequent notation has been posted:-
http://www.metoffice.gov.uk/hadobs/crutem4/
The operative adjective is incorrigible.
Pointman
I wonder how many plagues will be required before the code is set free?
:)
The underlying code has been long known:
CRUTEM4=CRUTEM3+rand()*(year()-2011)/10
Perhaps Harry might get a job there and put together a readme file with lots of code as well as comments, and which could be widely shared in due course. He must have been pretty fed up with the work quality and environment at CRU, so even the Met Office might look attractive to him.
Harry might have taken all the code out and just left the comments in, since that was the only way to get CRUTEM4 to do anything meaningful.
"our preference is that the underlying station data, and software to produce the gridded data, be made openly available"
Maybe after the 'barbecue summer' forecast, what they meant was the 'griddled' data.
"... due to changes in the processing methodology used." = new adjustments to the temperature data.
Hot griddled comments, Alive! Alive, oh!
==============
The most disappointing thing is that when the actual CRUTEM3 code leaked out because it was mentioned in one of the CRU emails it was nicely written and it was very easy to spot the odd bug that myself and another person had seen: http://blog.jgc.org/2011/11/anatomy-of-crutem3-bug.html
Hopefully this is just a temporary blip and the Met Office (and others) will release code as a matter of course.
I commented a few days ago that their "talking point"now, on both sides of the Atlantic, is basically, "we don't need no stinkin' data--the science is settled." This post doubles down on that, adding that "you don't need no stinkin' data either, so we won't bother giving it to you."
I assume that you mean the article which is titled, "how Scientific is Climate Science?" At URL:
http://www.informath.org/media/a41.htm
I recall that James Annan made an acerbic post about this at the time when the announcement was made about "septics" (his term) having to put up or shut up. Do they really want sceptics to fall into line? Or is this deliberate sabotage? or.....maybe incompetence...and I guess James Annan is happy that he lives in Japan and is funded there where they seem to tolerate serial losers.
Be prepared for this shutter as well
They haven’t processed further for some days as I anticipated a few days ago.
http://ocean.dmi.dk/arctic/icecover.uk.php
Its not going their way that’s all I can say it happens every time ice goes up like this. Expect a major downward adjustments in coming days. The team cannot accept this data!
They're just following the basic precept of climatology - Jones' First Law:-
Why should I make the data available to you, when your aim is to try and find something wrong with it?
A book on investing pointed out that, if a company's annual results are late in being published, it is a very bad sign "Bad figures always seem to take longer to add up than good figures".
In the case of the MO, "Bad" = "does not show evidence of rapid global warming".
There is a text file on the Met Office Crutem site that still has the old wording:
Martin A "A book on investing pointed out that, if a company's annual results are late in being published, it is a very bad sign "Bad figures always seem to take longer to add up than good figures".
In the case of the MO, "Bad" = "does not show evidence of rapid global warming"."
Back around 2008/9 when I used to eagerly await each months results as a new way to highlight to warmists that it wasn't warming, I certainly noticed that "bad" (as in low) results would take up to 1-2 weeks longer to come out.
But there was one February's result when I just waited and waited and waited and I think it was nearly a month late before we got the result out showing a massive drop in the monthly temperature (probably 0.3C). It was the coldest month in 14 years ... as I naively tried to get the press interested in the story. Only to find the Met Office had spent that month preparing a blitz of pro-warming non-science so that by the time the the figure came out the press were sick and tired of the subject and not willing to print another article.
Perhaps what was even worse, was that slowly over the next few years that February dip disappeared and the only way I could now know when it was would be to go and see if I still have the statistics for the period.
The Met Office still plans to release the code.
I remember thinking at the time that "our preference" seemed to leave a lot of wriggle room.
For example, as Foxgoose alludes to above, Phil Jones may 'prefer' to release his data, but this is overridden by a preference for it not being criticised by Steve McIntyre.
Has not the Met previously used commercial and/or third party agreements to deny FOIA requests? I am sure they would have 'preferred' to release that information.
Well, release it then. It appears to be sufficiently complete to be in use, so what's the problem?
Although I'd love to know how the output can be validated to any reasonable level of precision, on the basis that last week's Inshore Waters Forecasts for the West of Scotland couldn't even get the current conditions right most of the time. The actual forecast was complete bollocks.
Richard Betts, if you get a few minutes could you please take a look at the following?
Previously on this thread I posted the notation on the HadCRUT4 page regarding 4.2.0.0 Jan 13 data.
I am puzzled by the explanation; the original issue was with how the CRUTEM4 numbers affected the HadCRUT4 calculations. Not with the CRUTEM4 numbers themselves.
HadCRUT4.1.0.0 Jan 13 was +0.432C made up from CRUTEM4 at +0.891C and HadSST3 at +0.292C.
HadCRUT4.2.0.0 Jan 13 was +0.378C made up from CRUTEM4 at +1.182C and HadSST3 at +0.292C.
So with HadSST3 remaining the same, the puzzle was how an increase of +0.291C in CRUTEM4 could result in a decrease of -0.054C in HadCRUT4?
There may well have been a problem with CRUTEM4 data but I am at a loss to see how that could be responsible for the above?
The "new" "corrected" 4.2.0.0 set of numbers for Jan 13 referred to in the MO notation are now:-
HadCRUT4.2.0.0 Jan 2013 is +0.450C made up from CRUTEM4 at +0.935C and HadSST3 at +0.292C.
Still puzzled, could you please ask somebody to explain where I am going wrong?
It would seem the MO has some previous in this general area:
http://wattsupwiththat.com/2013/02/12/uv-shift-in-the-leaked-ipcc-report-more-inversion-of-the-scientific-method/
"The methodology used to generate the original Met Office SSU data remains undocumented.."
Presumably they were worried that somebody might find something wrong with it...
It would be good if it could be communicated to the general public that scientists are not as they might be led to believe reading temperatures etc. off a dial which keeps going up, but are in fact using a variety of data which all has to be "processed" in complicated ways to give an answer heavily influenced by what those scientists think the answer should be.
They almost can't help it - it is what they were taught as students; the best marks go to those whose experimental results best match the textbook answer. (I have seen this done when the experiment was incapable of producing the results.)
Richard Betts -
Unless there is some proprietary code involved, any reasonable modern software development methodology will allow for publication of the contents of a particular version with little more than the press of a button, or a few lines of a command script. I've been involved with million-source-line programs which are required to archive all the relevant source code for each released version; it isn't hard if any forethought has been given to the problem.
If the MO are unable, you could ask someone well-qualified in both software and climate such as Steven Mosher to arrange the matter -- probably cost no more than a few thousand pounds. Unless there's some commercially-protected code of course, in which case it was unwise to promise a code release.
'still plans'. Heh, stillborn plans. Richard, your credibility dies with the Met Office. You've been used.
==============