Thursday
Aug072014
by 
Bishop Hill
Bishop Hill Forecast = projection = scenario
Aug 7, 2014 Climate: Statistics Matt Briggs has a post addressing the argument that climatologists are not making forecasts, they are doing projections. This argument, which has always struck me as a master piece of obfuscation, seems not to hold up under scrutiny.
Perhaps afficionados of "projections" can explain why he is wrong.
Reader Comments (60)
dougieh,
I haven't read it all, but from what I have read of the WGI report, it appears to be a reasonable assessment of our current scientific understanding. I can't say much about WGII or WGIII. I don't know if that answers your question, but it's the best I can do.
Okay, fair enough.
A projection is a forecast you don't believe in.
dearieme,
Many a true word said in jest?
Briggs misses the point. IPCC's forecast have two parts: emission scenarios which can be chosen by policymakers and and projections of the climate that will result.
ATTP http://bishophill.squarespace.com/contributor/25239856 Aug 8, 2014 at 3:17 PM
If the goal is to try and understand our climate system and how it is influenced by various physical processes, then I would argue that they are scientific models.
By this definition, all the activities of every engineering discipline are scientific. If the goal is to try and understand our engineered processes and systems and how they are influenced by various physical phenomena and processes, then I would argue that they are scientific models.
However, a massive big however, here's where engineering and Climate Science differ:
Scientifically, you may not even care if they end up matching reality or not, because either way it will be interesting.
Matching reality, high fidelity matching between the models and the physical domain is the sole objective.
Dan,
Do you mean at every step of scientific research, or simply the ultimate goal? The point I was simply making is that if a scientist runs a model that does not match what they are trying to study, that - in itself - may be an interesting result as it tells them something. Of course, they would try to work out what was missing/wrong and rerun the model. So, in some sense, the goal of climate models is to ultimately match reality (assuming that the chosen emission pathways, solar variability, volcanic activity also matches what actually happened). Of course, we only have one planet and can't travel in time, so this process may take some time.
What many who argue that there is some big difference between engineering and climate science often fail to acknowledge is that in engineering much of the fundamental science has been done and testing and validating models is significantly easier since comparisons with different test cases is possible. As I mentioned above, if we had multiple planets and could travel in time, maybe climate science could be more like how some people perceive engineering. Since we don't and can't, that seems like a rather impossible expectation.
What many who argue that there is some big difference between engineering and climate science often fail to acknowledge is that in engineering much of the fundamental science has been done and testing and validating models is significantly easier since comparisons with different test cases is possible.
We are all aware of the difficulties associated with Verification and Validation of GCMs. What we do not understand is the continued insistence by Climate Scientists that it is (1) impossible and (2) not necessary. In stark contrast, we insist that it is (1) possible and (2) necessary. Continued insistence to the contrary by Climate Scientists solely on the basis of authority does not make the problems go away. I predict that in the long run significant resources will be expended on these issues.
Verification and Validation in Scientific Computing.
Fundamentals of Verification and Validation
Verification and Validation in Computational Science and Engineering
We've also never said, It's easy. In my career I've never insisted that I could not address a problem because it might be too difficult. Such an outlook in engineering is not conducive to continued employment. In fact, I've found that the more difficult problems are much more fun. We always leave the easier problems to the academy so they can fatten up the CV. Test cases, for both Verification and Validation, could be developed for the various models and numerical solution methods used in GCMs. And I'm almost certain that some of this work goes on within the development community. It seems that the results are not very visible.
Earth's climate systems are no different from any other inherently complex systems. Engineered systems generally are comprised of complex geometric boundaries within which inherently complex physical phenomena and processes occur. And many of these interact with the complex geometric configurations in ways that are significant to the response functions of interest.
All the fundamental science for both engineering and climate science has been done. It's all mass, momentum, and energy all the time. Modeling of physical phenomena and processes based on the fundamental science is what engineering does. Modeling of physical phenomena and processes based on the fundamental science is what Climate Scientists do.
Fundamental science explores physical phenomena and processes based solely on properties of the materials of interest. Previous states that the materials have attained do not ever enter fundamental science. If previous states are a part of an approach to understanding the responses of systems, then universality is instantaneously and forever lost.
In order to actually learn something useful from numerical experiments with models, an absolute first requirement is that the models have some relationship to physical reality. Experiments with parameter perturbations, which btw are not properties of the materials, of models and methods for which no requirement metrics relative to fitness for purpose have been specified and met, is not an approach for which learning is assured.
hmmm . . . can't do href here ?? My constructs work on my machine.
http://www.amazon.com/Verification-Validation-Scientific-Computing-Oberkampf/dp/0521113601/ref=sr_1_1?ie=UTF8&qid=1407756474&sr=8-1&keywords=oberkampf
http://www.amazon.com/Fundamentals-Verification-Validation-Patrick-Roache/dp/0913478121/ref=sr_1_2?ie=UTF8&qid=1407756589&sr=8-2&keywords=Roache+verification
http://www.amazon.com/Verification-Validation-Computational-Science-Engineering/dp/0913478083/ref=sr_1_3?ie=UTF8&qid=1407756589&sr=8-3&keywords=Roache+verification
Dan,
Is this true? I'm not aware of having seen this explicitly. I can think of two things that they might be meaning. If they're referring to their ability to do long-term projections (till 2100, for example) then it would seem clear that we can't strictly verify them until we get to that point in time. So, I agree that - technically - one can verify climate models. If, however, we want verify that they are able to reproduce changes to our climate over many decades, I don't see how it is possible in reality (at least we can't do so now). As far as 2 is concerned, even today climate models give a range of possible outcomes that can be regarded as some kind of confidence interval. Therefore, we can still use this information to inform policy. The only reason we might not is if there was a really good chance that reality will be completely outside this range. Given that the range is consistent with paleo evidence and consistent with very basic energy budget type approaches, this seems unlikely.
First let me clarify. I omitted the word Independent as an adjective to Verification and Validation.
For me V&V means Independent V&V; the presence of the adjective is generally assumed and frequently omitted.
The issue has been the subject of discussions on several blogs for at least the last decade; RealClimate, wattsupwiththat, Pielke Sr., Pielke Jr., Climate Etc., among others. It is frequently observed, and by many independent commenters, all who have experience and expertise in software development, and application procedures, that none of the GCMs have been subjected to Independent Verification and Validation. From the continuous basic model equations to the output of an application and associated response functions, nothing has been independently investigated.
Peer-review of manuscripts cannot even begin to address these important issues.
This paper by Professor Steve Easterbrook and colleague is frequently cited whenever the issues come up: http://www.cs.toronto.edu/~sme/papers/2008/Easterbrook-Johns-2008.pdf Independent Verification and Validation are not mentioned in the paper. There are very likely other papers by the UoT group on the same subjects.
The software engineering/development procedures and processes cited in the paper are Standard Operating Procedures ( SOP ) for all software development projects with which I am familiar. The procedures and processes are carried out by the developers of the software. Flat out plain boiler-plate grade SOPs. These SOPs are no substitute whatsoever for Independent Verification and Validation. All software the application of which is important relative to the health and safety of the public, all without exception, are always subjected to Independent Verification and Validation. The software, all aspects of the software, used in Climate Science is the singular exception, as far as I am aware.
Can you point me to a Web site, report, or paper in which any of the GCMs used in the IPCC ARs have been subjected to Independent Verification. Verification is a mathematical procedure; it is not comparisons with measured data. That is the meaning in all the citations I listed above. Verification is required to precede Validation.
As far as 2 is concerned, even today climate models give a range of possible outcomes that can be regarded as some kind of confidence interval.
The members of the ensemble used by the IPCC are present based solely on the existence of the GCMs. The ensemble has been described as An ensemble of convenience. No success metrics for any system response functions have been defined for which a GCM must have validated against in order to be a member. If it compiles it is a member.
Under this situation, how is it ensured that the range of possible outcomes and 'some kind of confidence interval' are in any way related to physical reality? The Consistent With chronicles at Pielke Jr.,s blog are an interesting read.