Friday
Feb252011
by 
Bishop Hill
Bishop Hill Science policy on data and code
Feb 25, 2011 Journals This from reader Lance:
The Feb 11 Science magazine (v331, p649) states a new journal policy--they will now require that authors archive not only their data on a website (copy to be held at Science) but also their computer codes!
Think of the time and exasperation that Steve McI could have been spared had the journal that published the Hockey Stick had this policy in place.
On the other hand, the world would have lost the "rattlingly good" Hockey Stick Illusion as well.
I almost think that Science has been listening to M&M, WUWT, and Your Grace.
The new policy doesn't appear to be online. It's certainly welcome.
Reader Comments (10)
This reminds me of a comment by Phil at: http://wattsupwiththat.com/2011/02/10/reviewer-a-responds/
Phil implies that satellite data used for the Steig09 paper is also not available, and in any case due to its vagaries is also is likely to lead result in an error of 3C rather than Steig's claimed 0.1C in his reconstruction. The implications of this appeared to be lost in the noise of the methodology discussion and debate whether the number of pages of Steig's review was 65 or 88. So I reposted Phil's comment the next day at (see below) where Phil (not to be confused with Phil with a full stop who also posts on the thread) then responded to a question from bernie but suggested that he would not be taking this any further.
I know the methodology flaws in S09 are terminal, but I'm still surprised that I have seen no more on the issues raised by Phil. Anyone here have any thoughts?
===================================
lapogus says:
February 11, 2011 at 11:22 am http://wattsupwiththat.com/2011/02/11/o%E2%80%99donnell-responds-to-steig/ (about 1/3rd down)
I am also very intrigued by what Phil wrote last night on http://wattsupwiththat.com/2011/02/10/reviewer-a-responds/ – where he suggests that the satellite data available to Steig (with potential error of 3C due to cloud cover data uncertainties) could not have permitted the claimed the resultant temperature reconstruction accuracy to 1/10th of a degree. I repost Phil’s comment below in the hope that its significance is recognised:
Phil says:
February 10, 2011 at 9:33 pm
From Response to Third Review A, page 8:
…the reviewer seems to misunderstand the difference between spatial and serial [auto]correlation.
From http://climateaudit.org/2011/02/07/eric-steigs-trick/#comment-254159:
Steig stated by email today that he did not see the Response to Reviewer A’s Third Review…
It seems to me that Dr. Steig should not have launched into a public criticism of O’Donnell et al., when he, admittedly, was not a statistician and apparently did not understand a subtle, but key, statistical distinction.
Futhermore, it should be emphasized that Steig, et al. have not been completely transparent with regard to Steig 2009. They initially promised to electronically publish “all” of their data, but, subsequently, they have withheld and are continuing to withhold the raw satellite data as well as the details of the cloud masking. I don’t think that this data will ever be disclosed, because, if they do disclose it, IMO any remaining credibility would vanish. However, I could be wrong. Here is why I think that Steig et al. may need to be withdrawn:
1. Steig et al. claim that their study is based on the satellite data (NATURE| Vol 457|22 January 2009, pg 462):
We use passive infrared brightness measurements (TIR) from the Advanced Very High Resolution Radiometer (AVHRR), which are continuous beginning January 1982 and constitute the most spatially complete Antarctic temperature data set. (emphasis and acronym added)
They reference: Comiso, J. C. Variability and trends in Antarctic surface temperatures from in situ and satellite infrared measurements. J. Clim. 13, 1674–1696 (2000) (Comiso 2000) and state as the only explicit change to Comiso 2000:
We have updated the data throughout 2006, using an enhanced cloud-masking technique to give better fidelity with existing occupied and automatic weather station data. We make use of the cloud masking in (Comiso 2000) but impose an additional restriction that requires that daily anomalies be within a threshold of ±10 °C of climatology, a conservative technique that will tend to damp extreme values and, hence, minimize trends. (parenthetical comment added)
2. Comiso 2000 states:
Except for 1979 and 1992, when infrared data for the entire year were processed, the derived temperature data that are currently available are only for a winter (July) and a summer (January) month during (1978-1997). (parenthetical comment added)
References to seasonal trends are made in O’Donnell 2010, Steig’s first review (Review A) and its response, in Review C and its response and in Review D. In no way should this comment be taken as a criticism of O’Donnell 2010 as the processed AVHRR data was provided by Dr. Steig without further clarification and without any gaps (Steig et al Antarctica ant_recon.txt is what I had downloaded – it may have a different name now). The reader should be cautioned that Steig et al may have processed the infrared data for all months of the year and for each year, instead of what Comiso 2000 states, but, given the reference to Comiso 2000 and the refusal to disclose the raw satellite data, this question is not frivolous.
3. Steig 2009 makes no mention of how much AVHRR data is lost due to cloud masking. Kato et al. 2006 (S. Kato, N. G. Loeb, P. Minnis, J. A. Francis, T. P. Charlock, D. A. Rutan, E. E. Clothiaux, and S. Sun-Mack, Seasonal and interannual variations of top-of-atmosphere irradiance and cloud cover over polar regions derived from the CERES data set, GRL, VOL. 33, L19804, doi:10.1029/2006GL026685, 2006) states (pg 3):
The mean cloud cover over Antarctica is relatively constant, ranging between 0.62 and 0.75 during all seasons.
Thus, it can be assumed that only 25% to 38% of the AVHRR data is retained after cloud masking. Furthermore, Comiso 2000 states as one of their conclusions:
Among the key results of this study are the following: (a) satellite infrared data provide spatially detailed maps of surface temperature in the Antarctic region with an accuracy of 3°C…
Thus, each datum of the satellite data matrix provided by Dr. Steig should have an individual uncertainty probably greater than 3°C, but certainly in whole degrees C, yet Steig 2009 asserts that there is statistically significant warming by calculating trends in tenths of degrees C with confidence intervals expressed in hundredths of degrees C. I would submit that such an assertion is highly questionable and should be believed only upon rigorous demonstration. (Once again, this is no reflection on O’Donnell 2010 as they specifically state in the response to Reviewer D that:
Because our expertise is with the mathematics, we prefer to limit our paper to the mathematics.
)
Given all of the above, I respectfully submit that Steig 2009 should be withdrawn in its totality as the claimed warming trends appear to be a fantasy, given the cloud masking data losses of about two thirds and the uncertainty for each remaining datum of at least 3°C according to Comiso 2000 (Dr. Comiso is a co-author of Steig 2009). I remain open to withdrawing this comment if Steig et al. can rigorously demonstrate that the total uncertainty of the satellite data doesn’t swamp the small warming trends that they claim (I would likewise say that any claim that there is a small cooling trend would also be a fantasy on the same grounds). Withdrawal of Steig 2009, however, IMO should not impact O’Donnell 2010 as they have conclusively demonstrated that the statistical methodology employed in Steig 2009 is erroneous and that result would survive in any event. In closing, I would like to congratulate the O’Donnell 2010 authors for their fine scholarship.
My only comment on O’Donnell 2010 is that they should have included somewhere the response to Reviewer D’s comment on pages 1-3 of said response that the O’Donnell trend was not statistically different than the Steig 2009 trend, as this may be a common misunderstanding when comparing two different trends. I thought the explanation by O’Donnell et al. was very instructive and it shouldn’t be buried in a review response.
========================
If the science cannot be replicated it isn't science. (data)
If the science cannot be validated it isn't science. (methodology)
If the science cannot be verified it isn't science. (models)
If the science cannot be falsified it isn't science. (logic)
I wish people would recognise the difference a single pluralising 's' makes.
It's computer CODE not CODES and it makes me so bloody mad.
If it does no get enforced it will be a waste of time , so lets see how it works in pratice to see if 'reasons' are used for some people not to need to do it.
ABP - okay, that is nice of you to share that. Just out of interest, do you have a view on fudge factors?
How about back datig the policy, giving "scientists" a year to provide ther data/code, or see their paper deleted.
Jones et al UHI paper based on missing Chinese records comes to mind
The new policy is, as noted, not (yet) in their online "information for authors." The editorial appears here, free registration required. The relevant paragraph is:
The editors describe their prior policy as "Science's policy for some time has been that 'all data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science'," but then write of "encouraging" authors to comply. We'll see if the new policy is requirement or merely encouragement.
If 'Nature' is any guide to publishing practice as distinct from formal rules then it doesn't look good.
See 'The Search for Impartiality' at eadavison.com/?p=11 (Section headed 'Without preconditions – ‘Nature’ magazine')
This is a big improvement. No doubt due to Climategate and criticisms from web bloggers. I would like to see a few more improvements.
1) Published papers should list the funding source in the abstract. If I'm paying for it, I would like to know. If Bigoil or a think tank is, I would like to know that also.
2) If it is publicly funded, then it should be publicly available in ~6 months. I resent paying for research as a taxpayer and then having to go through a paywall to read the paper. Science magazine makes papers freely available after 1 year but I don't know of any other journals that do.
3) Any paper referenced in the IPCC documents must be publicly availble for free, as well as the supporting data, methodology and code.
lapogus - what you write is dynamite.
Given all the who-haa that has gone on about this issue,
I urge you, if you have not done so yet,
to make a formal apeal to the journal to withdraw the article.
Surely, nothing else will do.
(I'm sure we are all "so over" rubbish shock horror papers being published, with great fanfare by the MSM. Enough of that!).