Saturday
Apr192014
by 
Bishop Hill
Bishop Hill Tol tiff
Apr 19, 2014 Climate: WG2 Brandon Shollenberger and Richard Tol are having an interesting tiff over some late changes to Tol's chapter of the Working Group II report. Shollenberger accuses Tol of promoting his own work and of changing the tenor of the chapter after the final review had taken place.
On Twitter, Tol says that Shollenberger has it all wrong:
the SOD figure has 7 studies (Tol 1/7), the FGD one has 18 (Tol 2/18): 1/9 < 1/7
that's not true: section was changed in response to comments and extensively re-reviewed after.
This should all make for a pretty good dust-up in coming days. If nothing else it suggests that despite the Interacademy Council's review of the IPCC's procedures, the way Pachauri et al go about their work is still wide open to criticism.
Reader Comments (41)
Well, no doubt there is a perfectly transparent audit trail so we will be able to resolve what happened easily.
Ha Ha.
Seriously, why do we have to treat this stuff as if it had been done properly when it plainly is all crooked? Why do they need to cheat?
The don't appear to have an audit trail for changes, or sign off by others when a version changes. For many years in my career I had the the uphill task of making Marketing, Sales, and General Management people understand that having good quality control meant you couldn't cut corners. They thought it was all too bureaucratic. Given the number of scientific papers that appear having gone through the internal university process and the peer review process and are destroyed on the internet within a few days it makes one wonder if the same slipshod quality control is used in the IPCC reports.
To comment, one would have to read Brandon's complicated post.
Heya. Thanks for highlighting this issue. Also, thanks for hosting the documents for the IPCC AR5. I obviously needed copies of those when looking into this issue. I originally linked to several documents on your server, but for some reason the links didn't work. It was weird. If you clicked on the link, you got an error page. If you copied and pasted the link, it worked just fine. I couldn't figure out why that was happening so I had to change the links.
On Richard Tol's tweets that you highlight, the first one is misleading. It's true if all you're interested in is the number of papers referenced in each figure. However, I think we should consider the fact the final figure, in its entirety, is from Tol's work. Whether his work makes up 11% or 14% of the work referenced in the figure, the figure is 100% taken from Tol himself. That's important because figures can be dramatically different yet reference the same papers. That means simple counts don't tell us much.
On the second tweet, I'm open to hearing an explanation of how I am wrong. However, I've provided a link to the comments on the chapter the section was in, and aside from Tol's, I can find none which call for the section to be changed. Also, as far as I can tell, after the Second Order Draft is undergoes "expert review," there's no more reviews of the chapters. All there is is governments reviewing the SPM. I assume authors of chapters can review changes to their chapters, but I don't believe that is what anyone would understand as changes being "extensively re-reviewed."
I've asked Tol for clarification on those points, and I'll happily update my post if I get any new information.
And shub, I would like to think commenting requires reading the post. Sadly, experience teaches me otherwise!
To comment on the disagreement, one would have to read the post. To comment on the transparency or otherwise of the IPCC process is another thing. To me, if the process was adequately transparent then Brandon and Richard would not need to argue about what happened. It would be there for all to see. The process has been criticised many times for convolution and obfuscation. Why do they need to cheat, again?
rhoda
For the same reason that Nixon lied (and when he caught himself telling the truth he lied anyway to keep in practice).
Or for the same reason that in Aesop's fable of the scorpion and the frog the scorpion stung the frog even though it meant his own death: "It's in my nature".
rhoda, I agree wholeheartedly (except maybe with calling it cheating). The IPCC has done a better job of being transparent than in the past, but it still leaves a lot to be desired. That's why I wrote this post. I wanted to highlight the fact the IPCC process still suffers from the same problems as before. I get some people may not like it because the changes I highlight go the "other way" this time, but if it's wrong when one person does it, it's wrong when anyone does it. Right or wrong isn't determined by the "message."
That's why I wouldn't call this a "tiff." Tiff implies the disagreement is petty. I think it's pretty serious if the IPCC is still allowing authors such latitude they can make substantial changes without review, especially if doing so promotes their own work.
Brandon, my oft-repeated 'why do they need to cheat' refers to the whole MO of climate science, not the IPCC process per se. However, the IPCC continues to obscure what happens to review comments and how they may be inadequately addressed then ignored. If cheating is too strong a word, let's just say 'What have they got to hide?'.
The Inter-Academy Council Report of 30 August 2010 on the UNIPCC- " All Working Groups should use the qualitative level of understanding scale on their Summary for Policy Makers and Technical Summary, as suggested in IPCC's uncertainty guidance for the Fourth Assessment Report. ...Quantitative probabilities (as in the likelihood scale) should be used to describe the probability of well-defined outcomes only when there is sufficient evidence. Authors should indicate the basis for assigning a probability to an outcome or event ( e.g. Based on measurement , expert judgement , and/or model runs)...... However such guidance was not always followed , as exemplified by the many statements in the Working Group II Summary for Policymakers that assigned high confidence , but are based on little evidence."
Can anyone identify where the Authors of the Fifth Assessment Report have filed their bases for assigning the " high confidence" levels etc? Are these available for scrutiny anywhere by the general public or are they kept under wraps? I thought that the IPCC had accepted the Inter Academy Council's findings.
Brandon
Occupying the moral high ground is all well and good and in principle I agree with you but the IPCC has cheated, exaggerated, lied (remember the "world-class" scientists that turned out to be environmental activists with barely a decent science qualification between them?) since Santer single-handedly re-wrote the important bits of the Second Assessment and turned the entire global warming argument on its head.
It's a bit late in the day to be taking Tol, or anyone else, to task for trying to reinstate a note of caution into the proceedings now, wouldn't you say?
Mike: Thanks for your thoughts on the moral high ground.
rhoda, I understand. I don't even disagree there was cheating here. I just don't care to make general claims about cheating as it'd require I decide who I feel is responsible for what. That takes a lot more than examining this single issue. It's worth doing, but it's something I'll do another time. For now, I'm hoping people can just agree the IPCC process didn't work as it should have.
Mike Jackson, not at all. Nobody outside your "group" will listen to your criticisms if they are partisan. If you accept what Richard Tol did, you basically tell people to forget what you said about Michael Mann with the TAR and Keith Briffa in the AR4. They'll just see all of it as partisan squabbling.
Besides, there's really no benefit to what Tol did. If your concern is for the report being accurate, and you think his changes make the report more accurate, then maybe you should praise him. That's not my concern. I don't hope and pray the IPCC report is accurate. My hope is people will recognize it is untrustworthy. Then, either the IPCC process can be fixed, or people can stop listening to it. Either way, the accuracy of a single section means little to me.
Besides, the moral high ground is the most effective tool. Not only does it make people more likely to listen to you, it gives you the opportunity to build bridges. Imagine if our host, Anthony Watts and Bob Ward all got together and said, "We disagree on a lot of things, but we all agree the IPCC process is flawed." What would other people who support the "consensus" say?
Their only options would be to agree with what skeptics have been saying (the IPCC process needs to be fixed), or they'd defend Tol's actions, and thus, his message. Either way, you win.
Somehow I doubt this discovery is due to Bob Wards hard work. I'm just wondering out loud here that because Tol dared to question the religion of Mann Made Global Warming (tm) who it was that brought this to Wards attention?
Seems to me that there are a lot of people working behind the scenes to discredit Tol and further more it looks like this is no different to the goings on behind the scenes BUT the difference here is that Tol has been branded a denier, there he has to be rooted out at all cost!
Mailman
Initially, the impact of climate change on economic activity was in Chapter 10 and the impact of climate change on economic welfare was in Chapter 19. This never quite worked. Chapter 19 also lacked the necessary expertise, so they enlisted me as a contributing author.
This bit of Chapter 19 was substantially rewritten and became nonsensical. For instance, a figure was included to show the impacts of climate change on different economic sectors according to DICE, a one-sector model.
As Chapter 19 was running out of time and pages, it was agreed to shift this part to Chapter 10. The comments on the Second Order Draft called for a complete revision, so that's what was done. The revised version was subsequently sent to a number of reviewers, revised again, and sent for the final government review.
The similarities between my work and the IPCC's are there because both discuss the population of studies -- all 18 of them. A textual comparison would reveal that the emphasis and style are quite different, though.
Mailman, I hope you aren't suggesting I am working behind the scenes to discredit Richard Tol. Aside from e-mailing a few people to alert them of the post I wrote (because it was related to things they had written) and responding to mail I get, I haven't done anything that wasn't in full view of the public.
As for Bob Ward, it may not have involved hard work, but I don't doubt he discovered this on his own. All he had to do was look at the section which discussed a topic he had been discussing a lot. It's no surprise he would. I'd expect anyone who spent much time criticizing Tol's work to do the same.
Richard Tol I'll respond in more detail when I get home and have easier access to references (I'm on a phone right now), but thank you for at least confirming the changes were made after the last round of public reviews. That confirms nobody who signed up to be a reviewer could have seen it or commented on it.
I am merely asking who is helping Ward because 1) an aweful lot of effort is being put in to discrediting one man because, seemingly, he questioned the religion of Mann Made Global Warming (tm), and 2) Bob comes across as being the kind of guy who has never had an original thought in his entire life.
You say you are just as tough on the religious followers for doing exactly the same thing that Tol is supposed to have done. It would be good if you could post a link or five to blog posts etc where you have taken them to task as well?
Mailman
Mailman, you sound like you're doing more than asking who helped him when you say it seems "there are a lot of people working behind the scenes to discredit Tol." Given this post is primarily about what I've said, it'd be easy to interpret your remark as referring to me. It might be worth being more clear on what you mean in the future.
As for me, I think it'd be unfair to ask for posts on similar issues. I've had a "blog" for only a few months. I started posting on it years after the last IPCC report came out. I can easily direct you to posts I wrote criticizing "the religious followers," but not on this particular topic. As an example, something like 15 of the 80 or so posts at my site have been in a series highlighting the biggest examples of wrongdoing by Michael Mann in his work (I still need to finish this series). I also wrote what, as far as I know, is the only substantial response to Mann's book.
I've also written multiple posts criticizing Skeptical Science, highlighting there use of a fabricated quote. This led to them discussing the issue in their forum but refusing to admit anything publicly (a fact I highlighted in another post). I also wrote a number of posts criticizing Stephan Lewandowsky, even going so far as to make a poll to give a practical demonstration of why his survey "results" are meaningless. This last one is important because, as far as I know, there had never been any explanation provided as to how he could find "statistically significant" correlations which didn't seem supported by the data.
Or to put it more simply, nearly everything I've ever written in the global warming discussion has been on the "skeptic side."
Alright. I'm home so I can respond with more detail. Richard Tol claims:
The figure in question is 19-8, and if you listen to Tol here, you'd think the figure only includes information from the DICE model. In actuality, it lists three different models, DICE 2007, ENVISAGE and FUND 3.8.
The next thing to note is Tol claims DICE is a one-sector model, thus it is nonsensical to claim the model shows damages across various sectors. This is a strange claim. If DICE couldn't show damages across multiple sectors, how did the IPCC get values from DICE for damages across multiple sectors? Unless Tol wishes to claim the data used in the Figure 19-8 was fabricated, I see no way for him to make this argument.
As best I can tell, Tol's argument is simply misleading people via semantics. IT's thinking of sectors in models as "boxes." A one-sector model would be a one-box model. One-box models can have many outputs. A one box climatic model could have temperature, pressure and volume as output. Similarly, a one box economic model could have damages to agriculture, energy supply, coastal property, labor force, ecosystems and many more thing as outputs.
At least, that's the best interpretation I could come up with. Regardless, one box/sector models can have submodels, and if we check documentation for the DICE model (such as here), we see mentions of many different sectors (other documentation gives more specific listings of such sectors). That, combined with the fact the IPCC had data from the DICE model on damages for various economic sectors should be enough for people to say Tol's claim is wrong.
I asked Tol what comments the changes were supposedly made in response to. He hasn't answered. However, I've read every review comment on the chapter, and I'm confident none "called for a complete revision" of this section. There were a few general comments suggesting tightening up language/condensing material within the chapter to save space, but that's about it.
Comments specific to the section are #770-#794. Three are from Tol himself, and they obviously don't count as showing what other reviewers thought. Seventeen were editorial comments or generic suggestions about making things more cohesive (making things mesh with other chapters). One asked for more discussion of uncertainty. Another suggested one paragraph be moved to another chapter. Another suggested a paragraph about biodiversity/species extinction was redundant.
That leaves comments #770 and #772. Commenter @770 said he likes the section but suggests some text might be condensed (but finds no specific recommendations). #772 is the only comment one could argue calls for any significant revision. Specifically, it does suggest economic and ecological damages be separated. That single suggestion is the only suggestion that supports Tol's claim in the slightest.
But even comment #772 doesn't support Tol's claim. #772 called for the section to be separated into two components, which it was. It did not, however, call for the material discussing economic damages be rewritten. Had Tol merely moved the discussion of economic damages to a different chapter (thus separating economic/ecological issues), I wouldn't have commented. That'd be a reasonable thing to do, and one reviewer did suggest it'd be done.
However, as far as I can see, none of the expert reviewers "called for a complete revision." If Tol or anyone else thinks otherwise, they ought to cite the comment numbers which support their claim. As it stands, it appears he has simply made this up.
@Brandon
"how did the IPCC get values from DICE for damages across multiple sectors?" is indeed the pertinent question. Not from DICE, that's for sure.
Richard Tol, I find it curious you've chosen not to answer my direct request for information supporting a claim you made. You said the "comments on the Second Order Draft called for a complete revision." I disputed this then asked you to support your claim (twice). I've also directly disputed this, pointing out the evidence does not seem to support your claim. You chose to respond without addressing what I said. This is awkward given how much of a fuss you have made about people hiding data. It's difficult to see how you could complain about some people hiding data yet refuse to provide information about references when asked.
As for the little you have said in response to me, I find this fascinating. You claim to be sure the IPCC could not have gotten the data plotted in its figure from the DICE model. That would suggest the data was fabricated or completely mishandled (such as using data from a totally different source than claimed). I'm going to leave it at that for the moment as there is plenty of other material to cover right now.
.
Specifically, people may be interested to know I uploaded a new post. In it, I explain how the section in question, which Richard Tol admits was heavily revised rather than simply moved (as he had previously claimed), gave prominence to a claim based entirely upon data from over a decade ago. In doing so, the section completely failed to discuss the results of work published since the last IPCC report, a task it was specifically required to do.
Tol's responses to it on Twitter have been priceless:
https://twitter.com/RichardTol/status/457759286027112449
https://twitter.com/RichardTol/status/457761002596368384
Maybe I'm wrong, but I don't think anyone will find his cursing at me to be a compelling argument.
Brandon,
I'm still waiting for your links to where you have taken to task anyone for doing what you accuse Tol of doing who supports the religion of Mann Made Global Warming (tm).
Btw, I'm not accusing you of helping off Ward...merely pointing out how convenient the clown is with his attacks.
Mailman
@Mailman
In fairness to Brandon, he does go after the likes of John Cook too.
Link or it never happened Tol! ;)
To be fair though, Cook is a minor player who is about as important as a limp wristed fish in a wet paper bag.
Regards
Mailman
@Brandon- "Nobody outside your "group" will listen to your criticisms if they are partisan".
Are you suggesting that the IPCC is not "partisan"?
So far I have found little to suggest that the IPCC is not partisan. Why should anyone listen to them (unless money is involved)?
Mailman, you can wait until the sun dies if you'd like. As I said, I've never written a blog post doing that, and it's unreasonable to expect me to provide a link to such a post given I've only had a "blog" for a few months. I can provide links to many posts I've written criticizing the people you refer to, but not on that particular issue.
If you want to see posts I've written criticizing the people you refer to on other topics, I suggest you begin with the list found here.
Don Keiller, no. I've said nothing of the sort. The closest I've come is I said people won't listen to your criticisms if the criticisms are partisan. A more accurate statement is people won't listen to your criticisms if they perceive your criticisms as partisan. People can be mislead.
Insofar as people view the IPCC as partisan, they won't listen to it. How accurately they can assess paritsanship is a different issue. In the case of Richard Tol abusing the IPCC process to promote his own work and views at the expense of all new information, the problem is incredibly obvious. That means people will easily spot any partisan defense.
In other words, if Tol wanted to abuse the IPCC process, he should have done it in a less glaring way. Then people would at least have some pretense for defending his actions.
There you go Brandon, that wasn't too hard was it?
Also, please remind me who the guy was who you hounded mercilessly when he dared question the religion over at Lucia's? Seems you do have form when it comes to these kinds of attacks.
Mailman
"if Tol wanted to abuse the IPCC process, he should have done it in a less glaring way"
let's change that to
"if Tol wanted to abuse the IPCC process, he would have done it in a less glaring way"
@Brandon "People can be mislead".
As Abraham Lincoln said "You can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time."
How true- the days of belief in the IPCC look increasingly time and population limited.
Mailman, I have no idea what you're going on about. If you're suggesting I should have provided links sooner, remember you asked for links to a specific thing. It should come as no surprise to you you don't get links to things you don't ask for links to. Also, when a person posts from a phone, it is often difficult to share links.
As for claiming I hounded someone mercilessly, I'm going to assume you're referring to my response to Richard Tol's ludicrous criticisms of Cook et al where he claimed patterns in sorted data proved Cook et al's results were nonsense. The only reason I hounded him mercilessly is that's an incredibly simple point, and it's obscene he's refused to correct it for half a year now. To be clear, his last response to me on the topic was:
Yet he had already acknowledged the data provided by Cook et al had been sorted by year published then title. It's impossible for me to imagine how one could feel a sorted data set is "a random reordering of a random draw."
Put simply, Richard Tol made a ridiculous argument, and I repeatedly pointed out it was ridiculous in the hope the error could be corrected. Failing that, I hoped people could at least acknowledge the ridiculous argument was ridiculous.
I don't think I deserve any criticism for that. I think the only criticism deserved is for people who won't say, "Finding patterns in sorted data doesn't indicate a problem. It's exactly what we'd expect.
@Brandon
I'm not going to go over this again. It did a placebo test. Standard practice and perfectly valid, whether you understand it or not.
I think perhaps the problem is all yours Brandon. The faked outrage is all there for everyone to see.
Either get over yourself or take a break and step out and get some daylight for a change.
Mailman
"Anyway, the onus is on you to show that a random reordering of a random draw should show a pattern"
Absolutely agreed.
The question is whether "does sorting by year behave like a random assortment (which it is, w.r.t the original order in which abstracts were rated), or, does it display legitimate time trends".
It is your claim that abstracts sorted by year show legitimate chronological trends. Why don't you prove/show it?
Just >saying< the trends seen in ratings sorted by year shows something won't cut it. Tol and I both believe ratings sorted by year are both random ratings, no less random than the original composite sequence. Whatever is seen in chronologically sorted data could just as well reflect bulk trends in literature, i.e., an increase in mitigation and adaptation papers.
It should be easy to do a multivariate logistic regression to check if chronological rating is independent of abstract category.
Mailman, I get you think the problem is all mine. What I don't get is why you haven't addressed anything I've said. Going after a person's character or motives to the exclusion of their arguments is the exact opposite of how a skeptic should act.
shub, I cannot make heads or tails of your response. You claim you and Richard Tol "both believe ratings sorted by year are both random ratings" yet say "chronologically sorted data could just as well reflect bulk trends in literature, i.e., an increase in mitigation and adapation papers." That means you're saying sorted data reflects a trend yet claiming it is random.
Richard Tol, what you did was not a placebo test. If it were, the authors must have been doing a placebo test when the published some of the same results you published.
For people unfamiliar with the topic, I'll provide a simple demonstration of the problem. Cook et al explicitly highlighted the non-randomness of their data when it's sorted by year. All three of the paper's figures were devoted to highlighting it. It was discussed in the abstract of the paper:
Cook et al discussed the non-randomness of their results in regard to time multiple times in their paper. They thought the point was so important they devoted every figure in their paper to it. Tol then found what the authors had repeatedly highlighted and argued it proved the paper was rubbish.
The argument could be fairly phrased, "Cook et al's figures prove their results are bogus!"
"Cook et al's figures prove their results are bogus!" is an excellent summary of Cook et al.
I honestly don't have a response to that. The best I can say is I think it says a lot that of the tens of thousands of people who looked at Cook et al's paper, it seems only Richard Tol noticed every figure in the paper proves the paper is bogus.
Actually, one other thought. Tol's initial comment regarding this argument was:
The data provided by Cook et al was in the same order as that presented by the Web of Science. Tol agrees the data was sorted by year (and title). To stand by his claim the order "is independent of the contents of the abstract," he must argue the year a paper is published has nothing to do with the contents of its abstract. I'd like to think nobody believes that.
I don't know about other people, but when I look at the figures in Table 3, I immediately see that there is something wrong with the sampling: two-thirds of the sample is discarded as irrelevant! What sort of the data collection strategy is that?
When I look at the figures in Table 5, I immediately see that Cook's claim of validation is, in fact, a claim of invalidation.
Comparing Tables 3 and 5 shows that Cook could not even have claimed validation as the subsample is unrepresentative.
I repeatedly referred to the graphs in Cook et al. I then suggested a hypothetical argument based upon those graphs. Richard Tol supported the argument. He now defends it by referring to things not included in the graphs. I get the word "figure" can be used in more than one way, but there is no way anyone could possibly read my comments and think I meant it the way Tol is now using it. This seems to continue the trend of him responding only to things I've never said like we saw on Twitter where he made a wildly untrue claim about what I've posted (then deleted the tweet to hide his mistake).
This sort of non-responsive response is all he ever offers. It's in the same vein as what else we've seen. For example, Richard Tol has refused to provide information (e.g. what commenters supposedly called for a total revision of the section) and he has provided false information (claiming the section was merely moved when it had been totally rewritten). To tolerate that while supporting his cries that Cook et al are refusing to provide information seems hypocritical.
@Brandon
Maybe you should develop a sense for when someone is pulling your leg.
As to the deleted tweet: I confused you with someone else, and I apologized as soon as realized my mistake. No need to let an obviously wrong statement stand.
Richard Tol, this isn't going anywhere so will you just answer one simple question: What "comments on the Second Order Draft" of the section we're discussing "called for a complete revision"?
Just check if consensus ratings, particularly '3' increase/decrease independent of category with time.
shub, if you have a point to make, please make it. One sentence comments which effectively tell a person to "look it up" are rude and contribute nothing.