Buy

Books
Click images for more details

Support

 

Twitter
Recent comments
Recent posts
Currently discussing
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« Library manoeuvres | Main | Kaye sera sera »
Monday
Dec142015

Superforecasting

I'm sure for many readers here, Philip Tetlock's famous studies of the predictions of experts need little introduction. His most famous finding - that specialists are considerably worse at making predictions than generalists - is now quite well known, although less so among the general public and media than it should be.

His more recent work has been a project looking at forecasts in the realm of geopolitics and the results are summarised in his latest book, Superforecasting. The project asked panels of volunteers to make forecasts and then tested how accurate they turned out to be in reality. Much of the book focuses on "superforecasters" - those people who turned out to be noticeably better at it.

Suffice it to say that there was little by way of expertise that made these people stand out from the crowd. They were bright people, but not off-the scale geniuses; numerate but not those inclined to conjure with arcane mathematics. It was more about independence of mind, the ability to constantly recalibrate and to question assumptions, the ability to think in terms of probabilities rather than in black and white terms.

And although the book barely mentions climate, and indeed the focus of the project was on short-term predictions not long-term ones, it was hard while reading it not to keep wondering "do climatologists think like this?"

Co-authored by pop-sci writer Dan Gardner, the text is as light as a feather, making it readily accessible to nearly everyone. A good Christmas pressie for the geek in your life. Buy it here.

 

PrintView Printer Friendly Version

Reader Comments (27)

So - a book effectively on why sometimes planes fall out of the sky but mostly they don't.

And nothing on climate change nor its supposed modelling weakness?

Ho hum.....

Dec 14, 2015 at 1:43 PM | Unregistered CommenterYouKnowNothingBishHill

Boy the trolls are keeping a sharp eye on the Bish these days. Trolls first up again. Must be getting under their skin!

Dec 14, 2015 at 1:47 PM | Unregistered CommenterAnthony Hanwell

Take 100 people.
Get them to make binary predictions on 5 topics.

On topic 1, 50 get it right.
On topic 2, 25 of them get it right
On topic 3, 12 of them get it right
On topic 4, 6 of them get it right
On topic 5, 3 of them get it right.

You now have three 'superforecasters'. Write a book about how they got it right. Under absolutely no circumstances allude to the fact that out of any random 100 people making five binary decisions, a random three of them will get them all right.

Dec 14, 2015 at 1:50 PM | Unregistered CommenterTheBigYinJames

TBYJ, if the result is the same as the coin-toss result you are correct. If the choice is one where judgement could have an influence, one expects some folks to beat the coin toss. If we run the test again and the same folks beat the toss, we have a meaningful result. Of course none of this has any predictive effect on the NEXT prediction they make. Or does it?

Dec 14, 2015 at 2:09 PM | Unregistered Commenterrhoda

Lady luck has no memory.

Dec 14, 2015 at 2:17 PM | Unregistered CommenterTheBigYinJames

Also as has so often been stated - modelling (including climate modelling) often doesn't do "predictions".

Start talking "projections" instead (you do know the difference?) and maybe you'll start to get the point.

Dec 14, 2015 at 2:17 PM | Unregistered CommenterYouKnowNothingBishHill

TBYJ, don't tell me you don't see the difference. This is not a coin toss. It is not a 50-50 ball. The events used in the test must be predictable. And of course a cascade test as you describe wouldn't be a good way to find anyone who can beat the general population consistently. It isn't roulette, it's poker.

Dec 14, 2015 at 2:32 PM | Unregistered Commenterrhoda

I get the point. Being wrong all the time doesn't disqualify you from being called an expert because in an obscure part of the paper you wrote 'suggest' and 'may' and it's not your fault if the university press dept. turns your guesswork into a probable or that gormless journalists then extrapolate that probable to a fact or that sanctimonious, faux-green rent-a-drones endlessly repeat this false fact in spite of numerous subsequent papers refuting it's methodology and assumptions.

Dec 14, 2015 at 2:39 PM | Unregistered CommenterJamesG

"It was more about independence of mind, the ability to constantly recalibrate and to question assumptions, the ability to think in terms of probabilities rather than in black and white terms."

That is a sceptic!

Usually older, well educated, working in science or industry, independently minded, numerically literate, a lifetime experience of seeing how easily it is to make bad decisions from bad assumptions.

Dec 14, 2015 at 3:01 PM | Registered CommenterMikeHaseler

I don't see sceptics as a monobloc. Some of us are like that.

Dec 14, 2015 at 3:02 PM | Unregistered CommenterTheBigYinJames

YouKnowNothing,, do climate projections have the ability to go down aswell as up, because climate predictions only ever go up. If this is the case, then projections may have about a 50% chance of not being as inaccurate, which would be a definite improvement.

Why didn't climate scientists think of this before? It seems so obvious to anyone who does not start with a false presumption.

Dec 14, 2015 at 3:03 PM | Unregistered Commentergolf charlie

Start talking "projections" instead (you do know the difference?)

projection (prəˈdʒɛkʃən) n

6. a prediction based on known evidence and observations

Or perhaps also relevant:

8. Psychology
a. The attribution of one's own attitudes, feelings, or suppositions to others:

In climate science, a "projection" is an estimate of the future which if actually comes true is hailed as a victory but if is nowhere close to reality is discarded and buried. E.g. claims of the Arctic being ice-free.

Dec 14, 2015 at 3:31 PM | Unregistered CommenterMikeC

In real life I find my ability tp predict is inversely related to how much I am involved with the subject and the outcome. The more dispassionate you are the more likely you can predict in a sensible manner. When it's your own future it's more unlikely you'll get it right. Every 'expert' fails this test if they have any involvement, financial or reputational.

How many alarmists predict the fiery death of the planet because they want to be the Cassandra who saw it coming?

How many scpetics predict nothing much will happen because they instinctively reject every alarmist story, because they can't contemplate ANY disaster?

Dec 14, 2015 at 4:01 PM | Unregistered Commenterrhoda

YouKnowNothingBishHill,
Fairly sure that I know what the IPCC means when it uses the term "projection". It's a simulation of the climate that could come true and so id testable.
A "prediction" is some incomplete aspect of the climate that can never be tested.

Strangely, "projections " are not to be used in making policies. Which is a bit of a blow for the people who just had a jolly in Paris.

But, please enlighten me, what do you think the difference is?
YouKnowNothingBishHill, please provide your definition of "Projection".

Dec 14, 2015 at 4:06 PM | Registered CommenterM Courtney

http://www.theguardian.com/business/2015/dec/14/spain-energy-giant-abengoa-collapse-predicted-by-17-year-old

Dec 14, 2015 at 4:24 PM | Registered Commentergeoffchambers

@ golf Charlie

Yes of course they do. What on earth are you thinking? Depends on input conditions...

@ MikeC

Honestly can't believe you had to look that up! Plus the definition is partial and not complete......

Dec 14, 2015 at 4:42 PM | Unregistered CommenterYouKnowNothingBishHill

YouKnowNothingBishopHill

40 percent youth unemployment in Spain after joining the Euro Nigel predicted that.

Dec 14, 2015 at 4:50 PM | Unregistered CommenterAl Gore doesn't eat off a paper plate

Here is most of a preliminary review by a statistician/economist involved in forecasting research himself, Frances Diebold.

Superforecasting is in the tradition of Nate Silver's The Signal and the Noise, but whereas Silver has little expertise (except in politics, baseball and poker, which he knows well) and goes for breadth rather than depth, Tetlock has significant expertise (his own pioneering research, on which his book is built) and goes for depth. Tetlock's emphasis throughout is on just one question: What makes good forecasters good?

Superforecasting is mostly about probabilistic event forecasting, for events much more challenging than those that we econometricians and statisticians typically consider, and for which there is often no direct historical data (e.g., conditional on information available at this moment, what is the probability that Google files for bankruptcy by December 31, 2035?). Nevertheless it contains many valuable lessons for us in forecast construction, evaluation, combination, updating, etc.'

The same chap has made some comments on a recent paper on Climatology and Predictive Modelling.

Dec 14, 2015 at 5:39 PM | Registered CommenterJohn Shade

YouKnowNothing, so after all the disastrous predictions, when do we get to hear of some valid projections? It just seems as though the predictions/projections of disaster, repeated ad nauseam in Paris, do not seemed based on any evidence at all.

Could you give a breakdown of what are valid projections, so it is easier to know what can be disregarded as speculative guesswork like Mann's Hockey Stick.

You are aware that all the ridiculous claims started with the global warming fantasists aren't you?

Dec 14, 2015 at 6:15 PM | Unregistered Commentergolf charlie

youknownothingtroll:

[...] modelling (including climate modelling) often doesn't do "predictions".

Start talking "projections" instead (you do know the difference?) and maybe you'll start to get the point

And this is just English semantics. Think about it. When COPout21 came to reduce the thousands of words of pre-amble into the 31 pages of press release, how many languages - with their own nuanced semantics - did they have to cover? Or was everything carried out in (UK) English? (Can't see that being the case - remember COPout was in Paris, France (for the benefit of our colonial cousins who might have thought it was in Texas) - and the French are very protective of their language. So, what I'd like to know is, what is the difference, if any, between "prédiction et projection" for them?

Dec 14, 2015 at 7:05 PM | Unregistered CommenterHarry Passfield

For US readers (who are well aware that Paris is in France, having made two 20th Century excursions there in large numbers to pull UK nuts from the fire):

http://www.amazon.com/Superforecasting-Prediction-Philip-E-Tetlock/dp/0804136696/ref=sr_1_1?ie=UTF8&qid=1450122816&sr=8-1&keywords=superforecasting

Dec 14, 2015 at 8:08 PM | Unregistered Commenterjorgekafkazar

Jorge.....Tongue and Cheek. No offence to my friends in the US.

Dec 14, 2015 at 8:34 PM | Unregistered CommenterHarry Passfield

Paris is indeed in Texas. I went there earlier in the year on a pleasant Sunday evening. It was shut.

Dec 14, 2015 at 8:43 PM | Unregistered Commenterrhoda

Rhoda, if Paris Texas is still shut, they may have derived some economic benefit from holding a climate party there. It would be just as destructive for the rest of the world.

Dec 14, 2015 at 10:15 PM | Unregistered Commentergolf charlie

Golf, I think the significant word in my comment is Sunday.

Dec 15, 2015 at 12:52 AM | Unregistered Commenterrhoda

Wasn't that the film with those folks in an ELdorado? The guy had snakeskin shoes.

Dec 15, 2015 at 10:20 PM | Unregistered CommenterMartin A

"Jorge.....Tongue and Cheek. No offence to my friends in the US." --Harry Passfield

None taken, actually. I've never been to Paris, TX, but have been to Swinney Switch. Unlike either Paris, not many people can say that. It's not easy to tell whether Swinney Switch is open or closed, since it looks pretty much the same either way. Odds are at any given day or time, it's closed, or might just as well be. However, it's not far from George West, the county seat, which has a stop light, and is just open enough on Sundays to serve food to truck drivers going to or from Corpus Christi on their way from or to San Antonio. It's untrue that people in George West gather at the stop light to watch it change color. I once read their entire phone book--five minutes shot to hell, but there wasn't much else to do there. Then I went looking for that stop light.

Dec 16, 2015 at 9:08 AM | Unregistered Commenterjorgekafkazar

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>