Bishop Hill Who's withholding what from whom?
Mar 7, 2010 Climate: CRU FOI There have been some interesting developments on the subject of which countries are preventing the release of their raw temperature data. You may remember that the Select Committee inquiry into CRU were told that several countries, among them Sweden, Canada and Poland, were refusing to allow him to publish these figures.
Anthony Watts is now reporting a press release by a Swedish pressure group called the Stockholm Initiative, who have obtained the correspondence between Jones and the Swedish Met Office, SMHI. The suggestion is that SMHI weren't in fact preventing release at all, but there has been a very long thread at Climate Audit where several people have disputed this. This posting is my attempt to make sense of it all.
In the weeks after Climategate, Jones wrote a letter to SMHI asking them for permission to release some data:
Given the importance of the global surface temperature series, we would like to make the underpinning data more widely available. CRU therefore requests your permission to release the underlying monthly mean temperature station records we hold - which for some stations go back to the 19th century...
We stress that the data we hold has arisen from multiple sources, and has been recovered over the last 30 years. Subsequent quality control and homogenisation of these data have been carried out. It is therefore highly likely that the version we hold and are requesting permission to distribute will differ from your own current holdings.
What seems clear from this is that what Jones wanted to release
- had not all been derived from SMHI in the first place ("arisen from multiple sources")
- had been adjusted ("quality control and homogenisation").
This clearly concerned SMHI, who refused permission for CRU to publish this adjusted data on the grounds "that the version of the data from the SMHI stations that you hold are likely to differ from the data we hold".
This refusal was then reported to the parliamentary inquiry into CRU. The relevant quote is this one from UEA vice-chancellor Edward Acton:
Professor Acton: Unfortunately, several of these countries impose conditions and say you are not allowed to pass it on, so there has just been an attempt to get these answers. Seven countries have said "No, you cannot", half the countries have not yet answered, Canada and Poland are amongst those who have said, "No you cannot publish it" and also Sweden. Russia is very hesitant. We are under a commercial promise, as it were, not to; we are longing to publish it because what science needs is the most openness.
However, among the papers released by the Stockholm Institute is a more recent letter, [update: dating from after the hearings] in which SMHI object to this characterisation:
It has never been our intention to withhold any data but we feel that it is paramount that data that has undergone, for instance, homogenisation by anyone other than SMHI is not presented as SMHI data. We see no problem with publication of the data set together with a reference stating that the data included in the dataset is based on observations made by SMHI but it has undergone processing made by your research unit. We would also prefer a link to SMHI or to our web site where the original data can be obtained.
Questions are being raised over the probity of the behaviour of Jones and Acton as revealed by these letters and I will try to make sense of these here.
It seems clear to me that Jones does not actually require permission from SMHI to release the adjusted data. This, by his own admission, is different to what SMHI holds and there can therefore be no issues of intellectual property. In this light, the refusal by SMHI looks odd, because it was not their place to prevent release of Jones' adjusted data. The question then becomes whether the UEA statements to the Parliamentary inquiry were reasonable. They indeed had a refusal from the Swedes, but this didn't relate to the data that everyone wanted to see - the raw data as used. We might note that it is not clear from the transcript which data is being discussed by the committee, although the exchange is in the context of a discussion of replication of findings, when the only relevant data is the raw data as used. I would therefore suggest that it was misleading of UEA to present SMHI's refusal as a barrier to making the data public.
Of course, later on, we can see that SMHI's refusual was in fact no such thing anyway. Having being asked for permission to release, SMHI felt they were being asked to endorse Jones' adjusted figures. Quite properly, they refused. It is clear that they had no objection to Jones releasing his adjusted data provided he made it clear that it was just that: adjusted. But to reiterate, this is a red herring. What is required is the raw data as used.
This doesn't look good to me.
Reader Comments (59)
I think you left out something really important: the letter where SMHI clarifies their statement isn't written until March 4:th, while the hearing was on March 1.
While climate scientists generally seem to believe in their ability to predict the future, so perhaps they should have known what SMHI would answer three days later, I don't think your summary is quite fair without this fact.
[Anna - I'd written "more recent", which I thought conveyed the meaning, but you're right it could be clear. I'll update]
I think the key word in Jones's letter is "underlying". To me it isn't clear whether he was asking permission to publish the original data he received from the Swedes (which may or may not have been processed by them) or his own adjusted data set and pass it off as the original. It is that latter possibility which probably agitated the SMHI. If so, it would have been prudent for them to try to clarify the issue before saying yea or nay.
It does seem as if the SMHI were saying last November that they didn't want Prof Jones to publish any altered data under the SMHI banner, but I can't be sure, and I think Prof Jones wouldn't have broke his back to further clarify once he got that magic "refusal" .
If the Pariament Chairmans question "Why did you hide it then?" is related to the pre-leak period, then Acton doesn't reply that "after" the leak occurred they later found that Sweden are "hesitant" to allow publishing, he rather leaves the impression they always knew this, seems a bit like they were scrabbling around to shore up their defence very early on.
Can we assume that Phil Jones wrote the same letter to all the sources of raw data? Does that mean that, with the exception of those who refused, all the ‘raw’ data held by CRU has also been through ‘quality control and homogenisation’?
Given that SMHI does it’s own homogenisation is it possible that the CRU have homogenised the data a second time at their end? Does the GISS publish the raw data; their own version of quality controlled and homogenised data; or the quality controlled and homogenised data as it comes from various sources such as SMHI? ie does the GISS raw data differ from SMHI’s quality controlled and homogenised data?
I’m beginning to think there must be something really scary about the genuine raw data for CRU to go to such lengths to hide it. This shell game Jones and Acton played with the Select Committee inquiry is very detrimental to their credibility.
Sorry, I was a bit hasty in my reading there, Acton didn't imply they already knew, since he said "...there has just been an attempt to get these answers."
My understanding is as follows. SMHI gave some raw data to CRU many years ago. CRU used that data to produce its gridded temperatures (CRUTEM). The raw data has since been quality-improved by SMHI. CRU is now being asked to release the original raw data, but CRU is bound by contract to not release that data. Thus CRU asked SMHI for permission to release that data. SMHI refused permission, but gave permission to release the quality-improved raw data.
Conclusion: Jones has done nothing wrong here; SMHI is confused and has confused others.
Doug
Where do you get the impression that SMHI improved the data? And is this relevant to the question of the raw data as used?
Isn't Jones saying that a part of his homogenised dataset had its origins in SMHI raw data, and hence he is doing SMHI the courtesy of asking them for their approval in releasing the full set?
SMHI then replies very clearly (I paraphrase) "Go ahead and make it available, but don't say it represents SMHI data, because you have homogenized it and hence it no longer agrees with our own figures."
Jones could only have genuinely seen that as a refusal if he didn't read the SMHI reply at all closely. Or perhaps he knew but didn't explain things clearly to Acton (who actually made the public comment.)
Again, it's a toss-up between incompetence and malfeasance.
I agree with your interpretation of events.
Sadly, Acton and Jones's responses are likely to be given the benefit of the doubt.
Clearly Jones asked SMHI permission to release fiddled-with data, because, in my opinion, he was looking for a rejection.
Bishop: "It seems clear to me that Jones does not actually require permission from SMHI to release the adjusted data".
That sound reasonable, but what is that agreement not to publish received data without permission. Is there a contract, and what is in it in this dimension? Is it completely empty?
From discussions elsewhere (WUWT) I got the impression that both the CRU and SMHI have their own homogenized data, but the two homogenizations are different and SMHI doesn't want the CRU's data to pass as SMHI data, if only to avoid confusion. OTOH, nobody is posting the raw data, which would be the interesting part, just the pasteurized ones.
Jones asks:
Which up to that point sounds something reasonably like CRU are asking for the raw values they already have from Sweden, (maybe "monthly mean" is is different from the raw too?), but then Jones goes on to to muddy the request by talking about his "multiple sources" and "homogenisation". I can therefore understand a responder in Sweden having noticed the Climategate noise (their response is a month later than the original 30th Nov request) then deciding not to get embroiled in a to-and-fro clarification process.
I would paraphrase that SMHI response of theirs as "No! Don't release any unknown altered data at your end which could attach us to this issue, Just use our original web-source where we know what the hell is going on."
I have followed this discussion since it started, and to my opinion the Swedish do not want to have the CRU homogenized (like a sausage?) SMHI data published as being SMHI data. They are not anymore after the magical processing by CRU (value added product). If I had provided data to CRU, I would not like to be associated with the product. I prefer to walk in the open, without having to hide my face in a brown paper bag.
Surely the key point is that SMHI never said he couldn't release the data.
They simply said: Don't associate us with whatever you release.
Your Eminence—
A letter from SMHI says "the version of the data from the SMHI stations that you [CRU] hold are likely to differ from the data we hold". My interpretation of that is that SMHI has quality-improved the raw data since originally sending the data to CRU. I am not certain of this though. It could also be interpreted as implying SMHI believes that (1) CRU no longer holds the original raw data and only has the CRU-homogenized data and (2) CRU wants to publish the homogenized data as though it were the raw data. If so, that belief is apparently incorrect.
In either case, this my suspicion is that there is just a miscommunication. Perhaps this can all be cleared up on Monday.
there is another issue here, in that in times past, CRU has claimed it has existing contracts that forbid it from passing on data.
This current exercise has got nothing to do with existing contracts that forbid it from passing on data. Instead, they are writing to NMS, and asking them for permission for CRU releasing data as of November 2009. It seems to me that a country with no existing contract could forbid the release of data on those terms.
If the choice is between cockup or conspiracy, I think its a 50/50 choice. Either is entirely credible where the CRU is concerned.
Why does someone not publish both the raw and homogenised data SMHI says is available along with the CRU version?
You ever met a jolly Swede? 'ave you? 'ave you? Miserable beggars the lot of 'em. Why don' they piss off an' commit suicide, an' stop messin' with my Phil? Bloody Nazis in the war, they was.
Looking at that SMHI 4th March clarification, to my mind it confirms my suspicion about SMHI's (reasonable) desire just to keep their distance from the whole climategate issue, hence, in their original 21th Dec response to Prof Jones letter they say:
I think that "Given the information" refers to Jones's muddying "homogenisation" and "multiple source" qualifications. It gave SMHI an early out.
Then, after seeing themselves indirectly getting a mention in the 1st March enquiry, they feel they have to clarify on the 4th March but only really by making the same point in a louder voice:
I think it is a bit or Sir Humphrey-esque way of rasing their voice and saying they don't want to be mis-characterised in any way
Does "original" here mean that this is the only data that CRU could have got from Sweden all those years ago? It seems that way to me.
It's never wise to overlook the snafu principle, and this may indeed be an innocent miscommunication. But it's one which turned out very conveniently for Jones and his minder, and given his form for cooking up ways to frustrate FOI requests, the alternative that he
Sent the Swedes a request, substituting the words "underpinning" and "underlying", for the unambiguous "raw", which he hoped and believed they would reject,
The Swedes, who are presumably on the 'net and had a pretty good idea of what he had been uo to, were horrified at the prospect of Jones' "value-added" data being published with their imprimatur, and obliged him with the refusal he needed to get him through the Committee hearing.
After the hearing, realising the role they had unwittingly played in Jones' exculpation, and the shadow it cast over their reputation, the Swedes wrote in haste to clarify their position, and tipped off the Institute to make sure the letter got published.
Will he get away with it? I got the feeling, watching the hearings, that Graham Stringer might quite like to hear this story....
In my opinion, Phil Jones never asked Sweden for permission to release their raw data because he made it quite clear that what he wanted to release was adjusted data, based on their raw data. This was what Sweden rejected. Therefore, the statement that Sweden denied permission to release raw data is incorrect. This is subsequently proved to be true in Swedens subsecuent communication.
I think that the case is black and white.
Er, because CRU won't publish it, saying it's not allowed to? ;)
Bishop - perhaps someone in the UK (I'm in Sydney) can put in a call to Stringer - details below - and just make sure he's aware of the Swedish correspondence. I've had a look at his background, and it's encouraging - grauated in Chemistry 1971, before Post-Normal Science came along and told scientists it was OK to lie in a noble cause - all reflected in his robust scorn for Jones in the hearings.
Party Name: Labour
Constituency: Manchester Blackley
Name: Graham Eric Stringer
Date of Birth: 17 February 1950
Personal Website: http://www.labour.org.uk/maps/locinfo.phtml?ctid=2326
Party website: http://www.labour.org.uk/
WestminsterTelephone: 020 7219 6055
Hopefully Jones and Acton perjured themselves, but things were worded in such a way as for them to squirm out. Can the Parliamentary Inquiry call them back in for questioning?
Are people answering questions in a Parliamentary inquiry under oath? If not, perjury doesn't apply.
I assume they are under oath. What's the point then? Why waste everyone's time and money?
Rick Bradford, Kevin—
I telephoned the Committee about this a couple weeks ago. I was informed that everything told to the Committee, verbally and in writing, is covered by parliamentary privilege. Hence, even if there were a falsehood (I doubt that), there could be no penalty.
They are not under oath. If they were then they would have been sworn in immediately prior to giving testimony. If the oath taking took place prior to them entering the room then they would have been reminded they were under oath.
Wow! So this was just a big charade? An attempt to appease the critics, but with no teeth? Pretty lame.
Jones has been leaning on confidential agreements with the various data source countries about not releasing data. There is very little said about that in this hearing. The best I have found are Q144 and 146:
"
Q144 Graham Stringer: According to Mosher and Fuller when you were asked to name, and Professor Acton has named a number of other ones, countries that you had confidential agreements with now, you could only produce the names of three countries. Is that right, when you were asked?
Professor Jones: I think it was about five.
Q146 Graham Stringer: Did you try to get round the agreements you had made with these different countries in the interests of scientific objectivity?
Professor Jones: Not in that way. We did, with the help of the Met Office, approach all the countries of the world and asked them whether we could release their data. We have had 59 replies of which 52 have been positive, so that has led to the release of 80% of the data, but we have had these seven negative responses which we talked about earlier, including Canada.
"
So to fix a poorely specified problem with five countries, if there was any, they approached all the countries. And that in the way we have seen in their exchange with SMHI, where it becomes clear that SMHI does not mind them publishing. Where are those original contracts keeping CRU from publishing their data?
"Where are those original contracts keeping CRU from publishing their data?"
They've lost it (according to Willis eschenbach), or never asked for it.
Also they've lost the original SMHI (and other?) data. Maybe they don't want to admit that they've delet... urm "lost" a helluva lot of important data?
Maybe it'll never be possible to obtain the raw data CRU used to homogenize their data.
On the issue of whether Jones and Acton mislead the committee I doubt that this could be proved. However, in order to get around the ICO saying there was "cogent prima facie evidence" that they had avoided providing data they needed to have proof that they were unable to provide it. I don't know if Dr. Jones is a complete and utter idiot or a smart political mover, but this refusal was a godsend to the UEA. He could easily have followed up with a request to point people to their website for the data, but stayed schtum. Idiot? Plotter? Who knows.
There's so much been going on recently but I seem to remember the UEA/CRU announcing they'd lost the raw data when moving office? Can anyone confirm that ?
This issue has led to an extremely lively debate at ClimateAudit. With respect to the references by Jones to Canada, I had a quick look at Environment Canada's web-site (ec.gc.ca) and quickly found its standard data licencing agreement. It is easy to understand and imposes few restrictions on redistribution of data: no fee can be charged, except for "value-added" data; credit must be given to Environment Canada; and the receiving party must agree to the same restrictions. The web-site has a wide variety of climate data, including historical data on temperatures.
An article on March 1, 1010 by Alex Morales at Businessweek.com referenced an e-mail from an EC spokesperson who stated that Canada releases temperature data to all requesters and all official data can be found on the web-site.
EC also does custom searches for $100 plus hourly fees for any additional processing and/or analysis. I was not able to find a standard agreement for those services, but perhaps I did not look hard enough.
Surely the simple explanation is Jones is trying to manufacture refusal from the Swede's to justify his earlier(false) claims that confidentiality obligations prevented him from releasing data. He must know that sooner or later he will be pressed on this point - 'show us the agreements' - so here he is creating one. Jones, not being the brightest boy in class (first degree University of Lancaster) doesn't realise that brandishing a refusal dated 2010 will not explain conduct circa 2007 - if thats when the claims that he couldn't release data were first made.
The parsimonious conclusion is that "Jones lied, data died." However much Swedish data CRU received, when and in whatever context, is immaterial to the fact that SMHI considered him and his entire Warmist coterie untrustworthy. Over a period of years, Jones' deceitful and manipulative UEA --allied with GISS/NASA, Mann's ESSC et al.-- for crass pecuniary reasons materially misrepresented scientific fact while bitterly opposing all attempts to set the record straight. Splitting PCBS hairs in self-important official inquiries recalls Gilbert and Sullivan's "Mikado":
"The idiot who in railway carriages scribbles on window panes,
We only suffer to ride in a buffer on Parliamentary trains."
Like flashing signal-lights, overwhelming evidence shows that it is well past time to sidetrack Climate Cultists' extraordinarily destructive propaganda exercise, deleting every vestige of their bad faith, false pretenses "data" root-and-branch. Finns, Swedes, and Russians ought to lead the way.
I think this is the tip of an exploding iceberg.
http://strata-sphere.com/blog/index.php/archives/12964
TinyCO2,
I think you have it dead on (which is what I propose in my post linked above). Look Jones is clear he wants to put the SMHI moniker on the data he wants to publish and he clearly puts in the CYA warning CRU processed the SMHI data to a new state.
SMHI caught the subterfuge and said 'no'. CRU has a problem here because they need processed data in order to keep their house of cards standing. There have been tons of posts on that region of Europe showing how the CRU/IPCC/GISS graphs show no resemblance to raw data (I linked one such example).
I also agree that form letter was sent to everyone, and they are all right now checking to see if CRU adjusted their data and kept their name on it.
This does not surprise me one bit given the history of incompetence and arrogance from this gang of fools. And it would also blow this scandal wide open and destroy AGW if nation's started calling foul on data manipulation. It will take a week before we find out (or less). Once this brush fire lights it is all over.
How do members of your Parliament enjoy being lied to?
It seems quite clear, The Swedes didn't want CRU 'fiddled with data' presented as their real base data. So they rejected the request on that point, not the issue of making their real raw data public.
The whole house of cards, trying to get the CRU 'fiddled with data', established as the worldwide baseline 'real data' is coming apart at the seams. The fraud was woven deep, and CRU had to get the 'fiddled with data' as the baseline, by conning the others with going along. But apparently some climate scientists realized what was being done, and they weren't willing to play along.
Busted ....
Here is an interesting-ish quotation I came across today, Sunday. It is attributed to President Ronald Reagan and I found it on page 377 of Brewer's Famous Quotation 2006 compiled by Nigel Rees and I wonder if it could be used to describe the AGW movement". An alimentary canal with a big appetite at one end and no responsibilty at the other.
Surely original raw data should be owned and held in whichever country it was created as a master copy. If copies of that original master are then passed on to various places around the world and the recipients independently modify the master data in essence they are creating new data. Therefore isn’t it the responsibility of the recipient who created the new data to hold ownership and the audit trail for their newly created data.
Here are the term of use of SMHI data (first in original then in translation):
3.2 Licenstagaren äger icke rätt att använda data eller produkter som tillhandahållits i enlighet med detta avtal för kommersiella ändamål och inte heller för utveckling eller framtagning av meteorologiska, hydrologiska eller oceanografiska värdeförädlade tjänster. Licenstagaren äger inte heller rätt att vidaredistribuera, sälja, överlåta eller annorledes överföra data, produkter eller dokumentation i obearbetad form till tredje part med mindre än att man erhållit skriftligt medgivande från SMHI.
3.2 The Licensee owns no right to use the data or products provided under this agreement for commercial purposes and not for development or production of meteorological, hydrological and oceanographic value added-value services. The licensee does not own nor authorized to redistribute, sell, assign or otherwise transfer data products or documentation without further processing to third parties unless the parties have received written permission from SMHI.
It is very clear (especially in the original) that restrictions concern only if data is not processed further, i.e. raw data.
I'm bothered by the indication that Jones no longer has the raw data from Sweden. This seems to be part of a pattern of not keeping an audit trail for their data manipulations. Also, it's consistent with his apparent reluctance to release the code used for the manipulation.
Guys, careful using the word raw when talking about station data.
Monthly averages of daily or twice daily temperature readings can not be raw data. The rawest form of the data is the log book of the person taking the readings. Even typing it up can introduce errors like transposing digits, transposing entries, repeating readings or entering data for the wrong month, wrong station, etc.
Then calculating a monthly average can introduce errors if you do it wrong - but worse than that it's an arbitrary period of time and you are throwing away the variability of daily readings. Note that you are also over-weighting February's readings.
You may argue a good case for these arbitrary calculations - but it's now cooked data.
Jones is busy obfuscating the distinction between the observations collected and the mucked about with version he inputs into CRUTemp. The distinction between underlying and raw was very effectively blurred in the hearing.
Let us bear in mind that the letter from SMHI on 21/12 was drafted in the full and fresh knowledge of the Climategate expose. SMHI would most likely have looked at the Harry_read_me file and realised that CRU's databases were in a mess. They would have been aware of the building pressure for Jones to reveal his data and code.
They would also have been aware of the emerging problems with GHCN's station database, and problems arising from miss-use by GISS.
At some point they decided to build their own website to release station data. In the light of the Harry_read_me comments about CRU's databases, I think they would be very concerned about CRU data being thought to be the same as theirs. Hence the tone of the letter of the 21/12.
When Jones made his statement to the inquiry, SMHI had no option but to call him out on it. However, the letter of 4/3 could also be interpreted as "sorry, don't want to rock the boat, but we are going to cover our backs".
It was indicated in testimony that 14 countries were approached about releasing data, it would appear that 7 were either quite happy to ignore or missed the distinction between raw observations and underlying data for his program.
This isn't just science, this is climate science.
This aspect of the whole kit and kaboodle underpins my scepticism. Originally it was the unwillingness to release the core data that raised my interest. Now it is the blurring of the core data with the cleaned up 'product' that makes me smell a rat.
I get the impression that Jones et al 2009 felt that releasing anything to sceptics would be harmful, and that we were not sufficently versed in their work and so would find faults.
I've reached the conclusion that the CRU were not bad, but they were convinced of their results. So nothing that contradicted those results was acceptable. Actually my reading of the emails and statements since convince me that the bogeyman is actually the bogeyMann and we will see at least Jones if not others fall on their swords through their own decisions to support that individual, who is clearly pushing an agenda outwith his scientific remit.
In short:
Jones to Swedish Met Office - May I release your data, duly fudged by me, by a process unknown to you?
Swedish Met Office to Jones - No you may not. That is not our data - the data that you took from us. We object to our data, fudged by you, by a process unknown to us, to be released as our data, because it is not.
Jones to Select Committee inquiry team and to Nature - See I tried to get them to agree to release "the data", but they wouldnt let me.
Select Committee inquiry team and Nature - We agree with you Jones, these pesky deniers, why dont they just take you at your word?
If CRU did not have proper version control of their data and code (according to leaked code), then what makes us think Jones can even know what actually are the 'original data'. That may be the real reason Jones is blocking this.
I have seen (and sadly done this) myself - without version control, you end up with dozens if not hundreds of versions of the data all over the place, and if you loose the coder - you end up resorting to looking at file creation dates.
On a related note - has anyone noted Prof Acton's statement that CRU has only 3 F/T employees? What the .... Did not Jones attract millions of pounds in funding? Have we not spent billions on Climate Research? And the key data underpinning all this and the survival of industrialized society is held by three blokes at East Anglia who don't have the time to version control this data? Arrrrggghh.
'......We stress that the data we hold has arisen from multiple sources, and has been recovered over the last 30 years..... '
Multiple souces?
Has not Jones also said that some data has been lost?
Is Jones aware of the provenance for all the data he does hold?
Is Jones aware of the provenance for any of the data he does hold?
Phil is buying time that's it.
I don't fall for "Poor Phil his so disorganized", and so on.
But I think that Phil Jones had realised that if we discover is plot,
that he falsified the data to put a global warming bias in all the data that he contol,
imagine the fiasco:
CRU and the University of East Anglia will have very bad reputation, including Acton,
UK governement will be "embarrassed" a lot,
IPCC wil get is last "gate" and fall forever.
Jones will lost his job his prestige and his money and maybe go to jail.
So tell me why everytimes Jones speak in public everybody seems more confused.
And we still don't see data and code.
Phil is bying time that's it.