This is a guest post by Mark Piney. It looks at the Institute of Physics seminar on science policy of a few weeks back. Although the seminar has already been reported on here, Mark brings up different aspects of the event, which I thought were interesting.
Perhaps 100 of us sat in a rather airless meeting room, early evening on Monday 4th February at the Institute of Physics (IOP) to listen to four speakers (Roger Pielke, Georgina Mace, Richard Horton and Jonathan Breckon) give us their take on evidence and policy. The event was chaired by James Wilsdon from Sussex university’s SPRU.
Seminar set-up
James Wilsdon introduced the event saying that, in the UK, the debate amongst policy people, and in Whitehall (apparently), about the role of evidence in policy, had been “hotting up”. Things discussed included the use of randomised control trials, seeking external sources of evidence, and using the network of chief scientists. Together with a wider call for a stronger use of, and role for, scientific evidence. He thought that the discussion had been ‘a tad simplistic’ and that,
…applying evidence is rarely straightforward and often the turn to evidence-based policy shifts the political debate and argument to what forms of knowledge, articulated and advocated by whom, actually count in the policy process…
Apparently policy people are concerned to debunk linear models, and to emphasise ambiguity, uncertainty and complexity.
Each speaker spoke for ten minutes, and then questions were put to the panel. The whole thing was recorded.
Three of the speakers were selling something. Roger Pielke was selling his book. Jonathan Breckon was selling NESTA’s capabilities. Richard Horton was selling his left-wing, medical, state-interventionist ideology. The only person not selling anything was Georgina Mace. She just told the truth as she saw it; so refreshing!
So what did the seminar reveal about the speaker’s attitudes to evidence and policy?
- Pielke was mixed and odd, but did at least mentioned democracy and used the word ‘trust’.
- Breckon was catholic in his vision, straightforward and optimistic, and mentioned Parliament.
- Horton came over as slightly bonkers, blinded by his righteous ideological zeal.
- Mace was clear, honest, and keen on science separate from advocacy.
Here’s the seminar highlights, by topic:
Evidence and politics
Pielke explained why Hurricane Sandy, which did so much damage to North East coastal USA in late October 2012, wasn’t really a ‘hurricane’, and why. The states at risk and the insurance companies had, in anticipation of the risk, developed a ‘hurricane deductible’ element of $25,000 per household policy. But this only kicked in if a hurricane was declared by the US National Weather Service (NWS). Sandy was described by the NWS as a ‘post-tropical cyclone, of hurricane strength’, so the ‘deductible’ didn’t kick-in. The insurance companies, and ultimately their customers, will pick up a bill of around $20 billion, not individual constituents or states. The NWS did this because it’s a governmental creature, and because of overt public pressure from several influential governors and senators, one of whom sat on the NWS funding committee (Pielke gives more detail here).
As Pielke said, ‘In this case the evidence didn’t matter. The (political) decision had already been made’. Politicians didn’t want the evidence to matter. The mistake was made years earlier, when the NWS was founded.
He didn’t explain where you would best put an institute that had the responsibility to formally describe an event as a hurricane, to insulate it from overt political interference. Wherever it was ‘put’, it would need an independent source of income.
Later Pielke was asked whether he thought that the evidence was strong enough to support the UK Parliament voting, almost unanimously, for the 2008 Climate Change Act. His answer was accurate, but flippant, ‘Yes, obviously, the Act passed’ (audience laughter). He reified the democratic process, and didn’t really offer any solutions (see later) if our representatives make mistakes – huge mistakes like the UK’s ‘longest suicide note in history’.
Horton is very politically involved; very. He started his talk thus:
Today is world cancer day. So it’s particularly appropriate to have a public health part of this discussion, because I’m going to talk about the birth of the nanny state, a wonderful moment in the history of public health, which took place in 1962 with the Royal College of Physicians report on smoking and health, and that really marked the watershed in health where evidence in public health came face-to-face with policy..
Notice his use of the term ‘nanny state’. He’s not using it pejoratively, he’s pro nanny state. He’s celebrating it (or maybe he’s being very smart and it’s post-modern irony?). It’s a particularly revealing in the light of what he said later in answer to a question questioning his ‘politics’ and independence (as long-term editor of the Lancet)
He went on to describe the “policy-cycle”; timely delivery of evidence that comes from trusted sources, with the media on-side, and at a point where the zeitgeist is right. Tricky stuff. The zeitgeist was right for a smoking ban, but not for other nanny-state actions, yet:
The best available evidence tells us that we should tax junk food, and tax junk beverages.
In one fell swoop Horton reduces all the multifactorial complexity of a cultural and social phenomena, to one (populist) cause. Notice how his simplistic nostrum leads directly to a call for governmental regulation and taxes. To yet more regulating of the (sub-text, not-to-be trusted) marketplace.
It’s the same mind-set, almost the same phraseology, as Ed Milliband, when he was ‘minister responsible for climate’, endlessly incanting, ‘but the science tells us’. It’s politics, but it goes under the guise of ‘science’, of ‘evidence’. They use science as if it was god-given truth. And if it is treated that way, doesn’t that make the creators of these truths, priests, with all the power and influence that follows?
But where does the meaning, the status of evidence fit into Horton’s policy-cycle? He tells us:
And then the last thing that matters is whether the evidence is true or not, whether it’s valid. All those other things … they are all anterior to the last bit, whether the evidence is true, or not true.
This is an ambiguous statement. If evidence comes last in the policy-cycle, what’s its status? Shouldn’t it ‘know its place’? Couldn’t this, doesn’t this, potentially lead to production of evidence that ‘supports’ the policy. Isn’t there a big risk, that evidence producers are tempted to give you the evidence they think you want? And what price evidence-based policy if this happens.
And Horton amplified his views of politics and science in this reply to a questioner who accused of him of being very pro state intervention – of being ‘Dave Spart’ (a fictional joke ultra-left character from Private Eye magazine) – and asked him why the science his journal produces should be trusted.
Here’s Horton’s reply, which didn’t deal with whether his Dave Spartism sometimes affected his editorial judgement, but made lots of revealing assertions:
You know, the NHS when it was first created was created not to be this monolithic employer and organisation, which had these Trotskyist tendencies, it was actually created for a very simple reason, it was actually created on the principle of freedom. The idea that if you could attack disease, then that will actually deliver freedom. Freedom from a whole range of problems, and anxieties that faced the population at that time.
So the reason that I believe in judicious intervention, based upon sound evidence is because it delivers freedom. And the reason why I’m so horrified when people refuse to accept evidence about potential state intervention. Tobacco is one example, alcohol is another. Taxing junk-food might be a third, is because they are denying the possibility of freedom to citizens of this country or it could be any country in the world.
It’s not about control. You totally, totally get it wrong when you say that. And that’s my failure, for failing to communicate it to you. So I apologise for that failure. Because it is exactly the antithesis of control. It’s about creating the conditions for people to live free lives. To fulfil their desires, their preferences, their choices. That’s the principle on which public health is based. It’s not about nannying. It’s not about control. It’s not about the state. It’s about giving you, and your partner and your family freedom.
Horton’s post-hoc NHS ‘history’: Horton’s first paragraph is a complete re-write of the NHS history. It was a treatment service from the start in 1948, based on the principle, ‘that good healthcare should be available to all, regardless of wealth’ link. If there was any freedom involved, it was the freedom of not having to pay for health care or prescriptions or equipment. There were elements of prevention, such as vaccination against polio and diphtheria, but wide-scale public health initiatives were not part of the NHS plan, and they still aren’t.
Horton’s ‘freedom’ rallying cry, invoking an NHS public health foundation myth, is a post hoc fiction. Why does he do this, and what does he get out of it?
Horton’s ‘freedom’: In his second paragraph Horton blithely claims ‘the science’ tells us that taxing junk food would bring us freedom, and he uses the word ‘denying’. This is a key trigger, a dog-whistle word. If you don’t follow and bow down before my ‘sound evidence’, you are a denier. And being a ‘denier’ you are fair game.
Horton’s state-banning brings freedom and choices: So state regulation, if not banning and control is all about ‘freedom’, but only if it’s medically agreed and sanctioned ‘freedom’. Desires are OK, as long as they are the right desires, preferences and choices. Choices deemed acceptable by the public health fraternity and sorority. To the well-salaried liberal-left academics who create the ‘sound evidence’. There must be a terrible temptation to ‘create’ the ‘sound evidence’. Which in turn can be corralled into a consensus, which justifies their existence, power and control. It’s what priesthoods do.
While, at the start of the meeting, he wants and likes the nanny-state, here he denies he wants it. And he’s liberal in his use of the word freedom. ‘Freedom’ is an inchoate political rallying cry, but what’s he rallying people to do, and to want? What’s Richard Horton’s programme for power? A technocratic dictatorship? Endless ‘nudging’ by Richard, and his mates? It’s not clear, but it is clear that he has no intention of getting a democratic mandate by convincing people with arguments and debate, i.e. taking the democratic route. His ‘freedom’ is to come through lobbying, legislation and laws that ‘make you’ free. Orwellian or what?
Science, peer-review and truth
Peer review has become fetishized by the media, science-based institutes, and scientific pontiffs. And it’s used as a weapon by activist groups to shut down debate, and bludgeon politicians and business into doing things, things that the activists want. Or by politicians, to bludgeon us.
The panel were asked specifically about science and fraud, and the role of peer review. Horton answered thus:
Peer review and journals – I couldn’t agree with you more, absolutely. I mean peer review is an utterly corrupt, ignorant stupid, mad system that we’ve created it’s just that we haven’t come up with anything better.
But let’s understand what peer review is:
- Peer review is not about checking the validity of data
- Peer review is not about reproducibility of data
- Peer review is a check on acceptability, acceptability in the scientific community
And that’s why I think you will still need editors, because our job is to be awkward. To say, even though you’ve got three or four peer reviewers who don’t like this, and don’t want it published. To hell with them, we’re still going to publish it. Because it still says something interesting. It looks interesting, even though it’s not accepted by this tiny group of people who we call peer reviewers.
Horton’s said similarly critical things about the status of peer review before. And he’s probably the most honest and forthright of editors, saying, quite clearly, that peer reviewed science isn’t necessarily valid, reproducible or true.
But here he was more revealing of his attitude as editor of The Lancet. He portrays the process as him being sometimes awkward, a free spirit, and to hell with it if the reviewers don’t like it, if it’s ‘interesting’.
So then it becomes a question of what Richard Horton finds interesting. On the face of it this is completely arbitrary, and down to his personal knowledge and prejudices. So what he believes, sometimes becomes central to what’s published. At least he’s honest about what he does. And it may go some way to explaining how Wakefield was able to publish his spurious research connecting MMR vaccination to autism. And other ‘edgy’ papers that have appeared in The Lancet.
Evidence, policy and evaluation
Jonathan Breckon quoted David Cameron as saying’ ‘On all the big issues that matter to Britain we are heading in the right direction, and I have the evidence to prove it’.
Breckon talked about government, local and national and charities, ‘flying blind’, and the need for standards of evidence. Apparently the Dutch have had a ‘Dutch Central Planning Bureau’ (since 1945), and all policy has be vetted by it. The UK isn’t planning such an Orwellian beast but the Cabinet Office’s is keen to get ‘What Works’ institutes set up to cover social policy, schools, police and crime, copying NICE (National Institute for Clinical Excellence), which is regarded as successful.
He emphasised that evaluation should be planned and built in from the start of a policy. This is rarely done, which makes finding out what works difficult, if not impossible. Another contributor (Paul Nightingale, SPRU), who has done lots of evaluation, wasn’t convinced, as evaluation of large projects is difficult, and often protracted, and out-of-date by the time it’s delivered. While this maybe true, it appears to be a counsel of despair. If you can’t evaluate, how do you know what works, and how do you know whether you’ve wasted your (others) time and money?
Pielke was more confusing, and confused, on policy and evidence.
He recognises that a lot of research is wrong, and misleading. At one point he cites work done by John Ioannidis (link), who Pielke said found that,
Most of what we think is true (in health and medicine) is just wrong … because of the macro-structures in science such as the file-drawer problem...not publishing trials which came out negative....
This is an incomplete summary of Ioannidis’s work. Yes, part of the problem is the non-publication of negative studies. But the problem is far more to do with fundamental organisational and institutional bias. Often the forces skewing the results are to do with funding and tenure. But they are also to do with beliefs, which is where advocacy really gets a grip and influences ‘knowledge production’. Pielke didn’t delve into the fundamentals of this structural bias, which is odd because he identified the need for stronger scientific institutions later.
Science and advocacy
Georgina Mace was quite clear on activism and science:
I think the real problem is when the research is too strongly directed. It’s too narrow, its goal oriented, around proving a supposition is true or false. It doesn’t really help anybody, and it doesn’t actually move the agenda on at all.
And Pielke too:
Experts, and expert institutions, have choices in how they relate to decision makers. And the choice to become an advocate and try and reduce the scope of choice that’s available, is a noble thing to do in a democracy. But it’s not the only role that experts can play.
There are times like drug safety, policy makers will ask an expert questions that can be resolved empirically. You want to hear what can the experts say about that . That is not compatible with also having a role of advocacy…’
Horton’s an activist, and proud of it, along, apparently, with many other public health medics and scientists:
On the activism, you know, in public health this is just part of the DNA of a lot of people in public health. They go into public health because they want to see improvement in the health of society. And so they see their research as an instrument in order to do that. And they see themselves as inherently activist. Not activist in terms of marching or joining a political party but in wanting to see the evidence that they are generating actually make a difference in policy and politics. So I don’t think in public health anyway it doesn’t seem anything unusual. I do take Georgina’s point about the dangers of it, but … If think that this is different in some sciences, different disciplines perhaps have different standards by which they judge activism. In public health it would be seen as absolutely completely normal.
If Mace or Pielke had been asked to comment on Horton, which they weren’t, I guess they would have warned Horton that he, and his compadres, could either be advocates, or scientists, but they couldn’t be both. And this is true. But Horton, and the activist ‘scientific’ hordes, are in too deep. Too much money and too many careers are on the line. And anyway, on ‘Planet Public Health’ the zeal for ‘the cause’ tells them that what they do is right, and true. True priests have a hot-line to god. Public health priests have a hot-line to health ‘truths’, and we must obey them.
Good evidence, which is true, that you can trust
It’s true that what people perceive as reality, is reality, for them. If this perception clashes head-on with how the world actually works, the world will win hands-down, every time. Reality will, eventually, bite you on the bum. Societies and individuals suffer if they make big decisions on the basis of false beliefs; on unreliable evidence.
Former Cabinet Secretary, Gus O’Donnell, along with all senior civil servants, says that the central role of the Civil Service is to be ‘fair, honest, decent and true’. But how does Gus, or any individual, or organisation, find out the truth? Where do you go for evidence you can understand, that’s reliable, that you can trust?
Pielke was the only speaker that used the word ‘trust’, and made a trenchant observation about climate scientists, and the IPCC:
The tendency, and I think we see this in climate science… instead of calling for stronger institutions, what many in the scientific community do is assert the essential nature of science as an authority itself, that doesn’t require stronger institutions. ‘You should believe us, we’re scientists, we’re experts, we know better, trust us’.
And that exhortation to trust, doesn’t always work when things don’t go right. And I’ll just give you a quick example to end. The only scientific institution that provides advice to government world-wide, that I’m aware of, that doesn’t operate with a conflict of interest policy, is the Intergovernmental Panel on Climate Change. Now, for a body that asserts its importance, I cannot see how they can seek to maintain the trust of folks like we here, if they don’t abide by the very simple standard that every other scientist in international government advisory panels hold themselves to’
His answer, as a ‘policy wonk’ (his term), is to push for stronger (more independent?) institutions. Given that some, like the Intergovernmental Panel on Climate Change (IPCC) were set up by a political organisation (the United Nations) to do a political job, and others have been more-or-less been completely corrupted by shed-loads of money, and far too many advocate scientists, it seems unlikely that this will work. But I guess it should be tried. Perhaps Breckon’s NESTA, which prides itself as being innovative, can find ways of sorting out the salvageable from the lost organisations?
While Pielke’s clear about scientists and institutions claiming trust based on claims to authority, or simply by position in the institutional firmament, when it comes to climate policy he seems to be virtually divorced from the evidence:
The science is plenty good enough to justify action. Where the failure lies is with people like us, policy wonks who have not been creative enough to come up with policy solutions that meet broad common interest ...’
And;
The one role that we seem to not value. The expert body that opens up options. Instead of telling policy makers what they should do, they tell policy makers what they could do. The climate-change issue is at a standstill not because we don’t have enough science, it’s at a standstill because we don’t have enough options
Is he really saying that the empirical scientific evidence linking carbon dioxide levels to the Earth’s temperature, is so strong and unquestionable, along with the model projections, that all policy people have to do is consider control and mitigation options? On the lack of an increase in weather-related damage he’s very clear (link, link), but far less so on claimed human-induced global warming. Maybe he feels he has to accept IPCC pronouncements on ‘the science’ as the gospel truth, despite knowing, for example, that the IPCC has no conflict of interest policy, and uses blatantly partial NGO people and ‘grey’ literature.
And those options, that there ‘aren’t enough of’, what’s he talking about? Mystical sources of ‘green energy’? Magical green economies? What? It really wasn’t clear. Maybe it’s all in his new book!
Conclusions: Elephants, trust and speaker score-cards
Chris Patten (Tory panjandrum, and current BBC Chairman) believes that the ‘existential elephant in the room’ in any discussion of the future, is human-induced catastrophic climate change. I think his elephant is imaginary, not existential, and to mix metaphors, the noisy climate king that has no clothes.
There are far more solid and real elephants to notice and to listen to. Old fashioned elephants such as truth, and honesty, and how we get a grip on reality and what works. We’ll be much safer and happier if we listen to these elephants. They’ve always been our firm friends.
So, what’s the answer to the seminar title question; What counts as good evidence for policy? It’s the same old answer; evidence that’s well warranted, systematic, coherent, and objectively and fairly collected (as much as possible). Evidence that is, as far as is humanly possible; true.
And this, ultimately, comes down to trust: whether you trust institutions (Pielke), and individuals (Mace) to have moral and intellectual integrity. Are they honest, fair, intelligent and unbiased? These are all human qualities for which there is no substitute.
And another human quality, and effect, follows. Once a person or organisation has lost trust, it’s usually gone for ever. They’ve blown it. IPCC and Horton are examples of organisations and individuals, partially/wholly blind advocates, creating and corralling the ‘evidence’ for their causes. They blew it years ago. The Royal Society, and other august institutions, are in severe danger of following suit (link, link). We live in strange times, strangely dangerous times.
The meeting was ostensibly about evidence and policy, but ultimately it comes down to ‘Who do you trust?’
We all have our own built-in ‘trustometer’. Based on what I heard at the meeting, this is what my meter measured, on a 1 to 10 scale: Mace 9, Breckon 8, Pielke 7 and Horton 1.
So, who do you trust?