The code

This is a new thread for updates on the analyses of the data and code freed from CRU.

Everybody, I'm sinking under weight of things to do here. I need you to post one or two line analyses of what you are finding in which bits of code. I'll transfer these to the main post as they come in. It needs to be in layman's language and to have a link to your work.

CRU code

• Francis at L'Ombre De L'Olivier says the coding language is inappropriate. Also inappropriate use of hard coding, incoherent file naming conventions, subroutines that fail without telling the user, etc etc.
• AJStrata discovered a file with two runs of CRU land temp data which show no global warming per the data laid out by country, and another CRU file showing their sampling error to be +/- 1°C or worse for most of the globe. Both CRU files show there has been no significant warming post 1960 era
• A commenter notes the following comment in some of the code:"***** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********"
• Good layman's summary of some of the coding issues with a file called "Harry". This appears to be the records of some poor soul trying to make sense of how the code for producing the CRU temperature records works. (rude words though, if you're a sensitive type)
• Some of annotations of the Harry code are priceless - "OH **** THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm hitting yet another problem that's based on the hopeless state of our databases. There is no uniform data integrity, it's just a catalogue of issues that continues to grow as they're found."
• CRU's data collation methods also seem, ahem, amusing: "It's the same story for many other Russian stations, unfortunately - meaning that (probably) there was a full Russian update that did no data integrity checking at all. I just hope it's restricted to Russia!!"
• Borepatch discovers that CRU has lost its metadata. That's the bit that tells you where to put your temperature record on the map and so on.
• Mark in the comments notices a file called resid-fudge.dat, which he says contains, believe it or not, fudged residuals figures!
• Mark in the comments notes a program comment: "Apply a VERY ARTIFICAL correction for decline!! followed by the words `fudge factor' " See briffa_sep98_d.pro.
• From the programming file combined_wavelet.pro, another comment, presumably referring to the famous Briffa truncation: "Remove missing data from start & end (end in 1960 due to decline)".
• From the file pl_decline.pro": "Now apply a completely artificial adjustment for the decline only where coefficient is positive!)"
• From the file data4alps.pro: "IMPORTANT NOTE: The data after 1960 should not be used. The tree-ring density' records tend to show a decline after 1960 relative to the summer temperature in many high-latitude locations. In this data set this "decline" has been artificially removed in an ad-hoc way, and this means that data after 1960 no longer represent tree-ring density variations, but have been modified to look more like the observed temperatures."
• From the Harry readme:"What the hell is supposed to happen here? Oh yeah - there is no )'supposed', I can make it up. So I have :-)...So with a somewhat cynical shrug, I added the nuclear option - to match every WMO possible, and turn the rest into new stations (er, CLIMAT excepted). In other words, what CRU usually do. It will allow bad databases to pass unnoticed, and good databases to become bad, but I really don't think people care enough to fix 'em, and it's the main reason the project is nearly a year late. " (see Harry readme para 35.
• James in the comments says that in the file pl_decline.pro the code seems to be reducing temperatures in the 1930s and then adding a parabola to the 1990s. I don't think you need me to tell you what this means.

Comments

[hfj]

Climate (or, the author of this blog):

You're a sub-intellectual with just enough pretensiousness to think people want to read what you have to say; though I doubt some braindead sheep DO want to read what you have to say.

[Borepatch]

Their comments say they lost metadata and couldn't find it:

Bear in mind that there is no working synthetic method for cloud, because Mark New
lost the coefficients file and never found it again (despite searching on tape
archives at UEA) and never recreated it. This hasn't mattered too much, because
the synthetic cloud grids had not been discarded for 1901-95, and after 1995
sunshine data is used instead of cloud data anyway.

This is from README_GRIDDING.TXT.

[Jabba the Cat]

@ hfj

"You're a sub-intellectual with just enough pretensiousness to think people want to read what you have to say; though I doubt some braindead sheep DO want to read what you have to say."

Your AGW shill grant coming up for renewal per chance?

[Vinnster]

Just wanted to say you are doing a great job. If I knew anything about code I would gladly spend all my free time helping.

[lukas]

I've worked in computational chemistry for a while... this is not surprising to me. They are scientists, not software engineers (and they think they are above asking a software engineer for help.)

[Jeff Wood]

My Dear Bishop, I imagine you will leave the opening comment up. It is perfectly illustrative of the cast of mind of one strand of AGW opinion: shrill, adolescent anger. I suspect it is there, half buried, in some allegedly scientific types.

[Bishop Hill]

Jeff

I wouldn't dream of snipping a work of art.

[Thortung]

This comment pops up in a few of the source files:

;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********

:-D

[AJStrta]

Very good. I discovered a file with two runs of CRU land temp data which show no global warming per the data laid out by country, and another CRU file showing their sampling error to be +/- 1°C or worse for most of the globe. Both CRU files show there has been no significant warming post 1960 era

Link: http://strata-sphere.com/blog/index.php/archives/11420

[Steamboat McGoo]

"You're a sub-intellectual with just enough pretensiousness to think people want to read what you have to say;..."

Would that be pretentiousness, Einstein?

[Sebastian Weetabix]

is "hfj" Frank O'Dwyer gone bad?? Whoever it is, thanks for the self-parody. You can't buy laughs like that.

[Jonesy]

I dunno about you guys, but I'm gonna open a used hockey stick shop... just imagine when all this Al-Gore-rithm shit hits the fan, the lines of angry Americans lining up to buy one!

Wonder who would get pucked.

[gravamen]

Wikipedia articles that should record this event in a neutral and balanced manner, with journalist-written sources (best to discuss on the talk page, don't just start editing directly):

http://en.wikipedia.org/wiki/Climatic_Research_Unit_e-mail_hacking_incident
http://en.wikipedia.org/wiki/Phil_Jones_(climatologist)
http://en.wikipedia.org/wiki/Michael_E._Mann
http://en.wikipedia.org/wiki/Hockey_stick_controversy

[Paul Z.]

See this email conversation involving Ian (Harry) Harris, Tim Osborne, and Phil Jones -- another great example of "consensus" science:

http://www.eastangliaemails.com/emails.php?eid=1009&filename=1252090220.txt

If this isn't cooking the books, then I don't know what is. Since when does scientific data have to "look good" and scientists need to "be happy with the version we release"? Also, what the hell is a "IDL thingummajig"? Some magic toaster used to make climate change guano?

Just report the facts, that's all we ask.

[Steamboat McGoo]

When the whole truth finally comes out, I'll bet we find that "hacking" wasn't involved.

It had to have been either an insider (with a couple of login IDs) swiping (or even collecting and zipping the contents of) the 62 mb zipfile, or someone accidentally finding it on an unprotected server. Subsequent posting onto RC and the Russian server was done through an anonymous server - a simple technique widely used by lots of (non-hacker) folks to obscure their identity.

But CRU and RC will continue to spin it as "hacking" because its to their advantage to do so...

[Paul Z.]

Gravamen, it's really hard to edit wikipedia articles related to climate change because the AGW have an army of wiki editor gatekeepers to come up with some bullshit reason to remove what you've written. I've experienced this so many times. Just try writing something on the IPCC or Al Gore pages, you'll see what I mean.

A better strategy is write a google knol on the subjects you have outlined @:
http://knol.google.com/k

No editorial censorship from AGW zealots.

[Squidly]

But CRU and RC will continue to spin it as "hacking" because its to their advantage to do so...

I believe you are absolutely correct. I have been working in the computer science field for almost 30 years now, and this does not smack me as being "hackers" (I used to be one).

I am not buying into a word that Gavin Schmidt says about it. I do not believe RC was ever "hacked" or that there was even an attempt made. It simply doesn't make sense to do so and would have been a waste of time with unnecessary exposure and risk. Just doesn't make any sense.

RC is simply trying to play the victim card, in a broad CYA attempt. Its simply a diversionary tactic.

[Borepatch]

Experimental results of running the CRU code* here.

*Tongue firmly planted in cheek.

[mark]

More gems from the code:

From the programming file called "briffa_sep98_d.pro":

yyy=reform(compmxd(*,2,1))
;mknormal,yyy,timey,refperiod=[1881,1940]
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,'Oooops!'
;
yearlyadj=interpol(valadj,yrloc,timey)
;

November 24, 2009 | mark
From the programming file "combined_wavelet.pro":

restore,filename='combtemp'+regtit+'_calibrated.idlsave'
;
; Remove missing data from start & end (end in 1960 due to decline)
;
kl=where((yrmxd ge 1402) and (yrmxd le 1960),n)
sst=prednh(kl)

November 24, 2009 | mark
From the programming file "testeof.pro":

; Computes EOFs of infilled calibrated MXD gridded dataset.
; Can use corrected or uncorrected MXD data (i.e., corrected for the decline).
; Do not usually rotate, since this loses the common volcanic and global
; warming signal, and results in regional-mean series instead.
; Generally use the correlation matrix EOFs.
;

November 24, 2009 | mark
From the programming file: "pl_decline.pro":

;
; Now apply a completely artificial adjustment for the decline
; (only where coefficient is positive!)
;
tfac=declinets-cval

November 24, 2009 | mark
From the programming file "olat_stp_modes.pro":

;***TEMPORARY REPLACEMENT OF TIME SERIES BY RANDOM NOISE!
; nele=n_elements(onets)
; onets=randomn(seed,nele)
; for iele = 1 , nele-1 do onets(iele)=onets(iele)+0.35*onets(iele-1)
;***END
mknormal,onets,pctime,refperiod=[1922,1995]
if ivar eq 0 then begin
if iretain eq 0 then modets=fltarr(mxdnyr,nretain)
modets(*,iretain)=onets(*)
endif
;
; Leading mode is contaminated by decline, so pre-filter it (but not
; the gridded datasets!)
;

November 24, 2009 | mark
From the programming file "data4alps.pro":

printf,1,'IMPORTANT NOTE:'
printf,1,'The data after 1960 should not be used. The tree-ring density'
printf,1,'records tend to show a decline after 1960 relative to the summer'
printf,1,'temperature in many high-latitude locations. In this data set'
printf,1,'this "decline" has been artificially removed in an ad-hoc way, and'
printf,1,'this means that data after 1960 no longer represent tree-ring
printf,1,'density variations, but have been modified to look more like the
printf,1,'observed temperatures.'
;

November 24, 2009 | mark
From the programming file "mxd_pcr_localtemp.pro"

;
; Tries to reconstruct Apr-Sep temperatures, on a box-by-box basis, from the
; EOFs of the MXD data set. This is PCR, although PCs are used as predictors
; but not as predictands. This PCR-infilling must be done for a number of
; periods, with different EOFs for each period (due to different spatial
; coverage). *BUT* don't do special PCR for the modern period (post-1976),
; since they won't be used due to the decline/correction problem.
; Certain boxes that appear to reconstruct well are "manually" removed because
; they are isolated and away from any trees.
;

November 24, 2009 | mark
From the programming file "calibrate_mxd.pro":

;
; Due to the decline, all time series are first high-pass filter with a
; 40-yr filter, although the calibration equation is then applied to raw
; data.
;

November 24, 2009 | mark
From the programming file "calibrate_correctmxd.pro":

; We have previously (calibrate_mxd.pro) calibrated the high-pass filtered
; MXD over 1911-1990, applied the calibration to unfiltered MXD data (which
; gives a zero mean over 1881-1960) after extending the calibration to boxes
; without temperature data (pl_calibmxd1.pro). We have identified and
; artificially removed (i.e. corrected) the decline in this calibrated
; data set. We now recalibrate this corrected calibrated dataset against
; the unfiltered 1911-1990 temperature data, and apply the same calibration
; to the corrected and uncorrected calibrated MXD data.

November 24, 2009 | mark
From the programming file "mxdgrid2ascii.pro":

printf,1,'NOTE: recent decline in tree-ring density has been ARTIFICIALLY'
printf,1,'REMOVED to facilitate calibration. THEREFORE, post-1960 values'
printf,1,'will be much closer to observed temperatures then they should be,'
printf,1,'which will incorrectly imply the reconstruction is more skilful'
printf,1,'than it actually is. See Osborn et al. (2004).'
printf,1
printf,1,'Osborn TJ, Briffa KR, Schweingruber FH and Jones PD (2004)'
printf,1,'Annually resolved patterns of summer temperature over the Northern'
printf,1,'Hemisphere since AD 1400 from a tree-ring-density network.'
printf,1,'Submitted to Global and Planetary Change.'
;

November 24, 2009 | mark
From the programming file "maps24.pro":

;
; Plots 24 yearly maps of calibrated (PCR-infilled or not) MXD reconstructions
; of growing season temperatures. Uses "corrected" MXD - but shouldn't usually
; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.
;
if n_elements(yrstart) eq 0 then yrstart=1800
if n_elements(doinfill) eq 0 then doinfill=0
if yrstart gt 1937 then message,'Plotting into the decline period!'
;
; Now prepare for plotting
;

November 24, 2009 | mark
From the programming file "calibrate_correctmxd.pro":

;
; Now verify on a grid-box basis
; No need to verify the correct and uncorrected versions, since these
; should be identical prior to 1920 or 1930 or whenever the decline
; was corrected onwards from.
;

November 24, 2009 | mark
From the programming file "recon1.pro":

;
; Computes regressions on full, high and low pass MEAN timeseries of MXD
; anomalies against full NH temperatures.
;
; Specify period over which to compute the regressions (stop in 1940 to avoid
; the decline
;
perst=1881.
peren=1960.
;

November 24, 2009 | mark
From the programming file "calibrate_nhrecon.pro":

;
; Calibrates, usually via regression, various NH and quasi-NH records
; against NH or quasi-NH seasonal or annual temperatures.
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline that affects tree-ring density records)
;
perst=1881.
peren=1960.

November 24, 2009 | mark
From the programming file "briffa_sep98_e.pro":

;
; PLOTS 'ALL' REGION MXD timeseries from age banded and from hugershoff
; standardised datasets.
; Reads Harry's regional timeseries and outputs the 1600-1992 portion
; with missing values set appropriately. Uses mxd, and just the
; "all band" timeseries
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
;

[Led]

Hmm.

It's so difficult to make judgement calls on this issue but I do welcome the debate. I am rather sick of the high profile deniers, and to be honest I am occasional revolted by green scientists not being able to discuss the remote possibility of some (minor or possibly major) issues with the data.

Surely having more information in the public domain cannot be a bad thing. We paid for the research I'd like to see a little bit more than a graph at the end of it