Bishop Hill Research funding as the enemy of innovation
Sep 16, 2011 This is a guest post by Demetris Koutsoyiannis
I fully endorse Donald W. Braben’s statement, posted in Times Higher Education and linked in Bishop Hill’s post, that the current research councils’ system for selection of proposals for funding is
fundamentally flawed and a pathway to mediocrity.
I wish to offer an example and a suggestion for an alternative procedure. My example tries to offer additional evidence in support of the above statement. My suggestion is based on the approach Mother Nature follows in her selection procedures. These procedures always involve randomness, whose power has increasingly been recognized (cf. genetic, evolutionary and simulated annealing algorithms in optimization).
A. The example
I have put online my research proposal to the European Research Council—ERC (actually its first part, which was reviewed) along with the anonymous comments of panels and individual reviewers (I have not made public the signed documents as those who signed might not want me to publish their documents). My proposal was rejected twice by ERC, in 2008 and in 2011.
The interested reader can access and assess the proposal and its reviews himself. In my reading, there is a marked improvement in the reviews of 2011 with respect to those of 2008, the comments of which were rather general-purpose “copy/paste” text. However, the result was the same in both cases—rejection of the proposal. Of course, the panel’s recognition that:
The PI is extremely well known internationally, with an excellent publication and citation record and important international scientific responsibilities, and is recipient of prestigious awards (Review Panel).
is of little solace. Some of the reasons the panel invokes for rejection, in my view are reasons justifying funding of my proposal. I quote below a few examples followed by my comments:
Many of the statements are more philosophical than technical, and no details are given on how the different activities will be connected (Review Panel).
The aversion of the Panel to philosophical issues and its clear preference for technicalities and details are really saddening.
While ambitious and potentially important, the PI bases none of this research on current understanding and wishes us to accept that brand new thinking is required in all aspects of the proposed work. I find it difficult to accept that decades of climate change and hydrologic research have proven nothing useful (Reviewer 2).
No, I do not wish this reviewer to accept the necessity of brand new thinking; I would not even try to convince the reviewer that research is all about improving and often abandoning current understanding. But I wish the ERC did not use rhetoric that is inconsistent with its practices. It is deceitful, on the one hand, to announce encouraging of groundbreaking research and, on the other hand, to use reviewers with such convictions about research.
[T]he proposed approach based on "a novel mathematical framework to quantify uncertainty in nature" including the development of "a new hydroclimatic theory" is not, as described, fully convincing (Reviewer 3).
Would a proposal whose summary (because just the summary was evaluated) was fully convincing for a typical reviewer really justify funding? Can ground-breaking research be fully convincing at the point of its announcement in the form of a summary?
The level of research funding seems patchy – despite the statement of “generous funding from Greek authorities” it seems that current funding is small (Reviewer 4).
In other words, only those who already have sufficient funding are eligible for funding. What an argument!
First, in order to promote one’s own research area it is not necessary to attack other research areas, especially when that attack seems unjustified (Reviewer 4).
It seems as if the reviewer dislikes attacks on established ideas. Can research be groundbreaking without attacking some established wisdom?
A second, and more serious, concern is the lack of specificity within the proposal, both concerning the tools that will be used and the data that will be exploited.
So now the reviewer has made crystal clear what innovative research is: it is a collection of tools and their application on data, provided that the collection and application have an appropriate level of specificity and, of course, the proposal does not make attacks. Furthermore, this research can be funded provided that the PI has already a large number of ongoing funded projects.
B. The suggestion
My suggestion for a better system is very simple and includes the following three steps:
1. Apply an initial screening of the proposals submitted, to exclude those which have been submitted for fun, those prepared by random text generators, those overusing clichés and those copied from existing documents (today there are reliable tools available to check the latter).
2. From the remaining proposals select those to be funded by lot. Lottery is a reliable system; it is used by Nature in evolution, and it was intensely used and highly appreciated in the Athenian democracy. Of course the details of the lottery need to be carefully studied. The system could assign prior probabilities based on measurable and objective criteria in order, for example, to penalize allocation of all funds to the same persons or consortia and to reward productive research efforts in the past.
3. After commissioning of the project, perform controls that the funds were allocated correctly and that the research produced accountable results (if not, ask for the money back).
I contend that such a system is clearly superior to the current systems applied by research councils worldwide, because it will not block novelty and innovation, which naturally attack conventional wisdom and establishment (cf. D. W. Miller, The Government Grant System: Inhibitor of Truth and Innovation?, J. Inf. Ethics, 16(1), 59-69, 2007).
Reader Comments (22)
Ironically, the "suggestion" is clearly too groundbreaking to be acceptable. It'll fall on the proverbial deaf ears - Review Panels rather enjoy their life and death decision powers.
We should push instead to have 20% of the funding allocated by lottery among the rejected (and serious) proposals. This will have the added bonus of comforting the Review Panels that they'll be keeping much of their powers anyway.
Pareto rules!
I have had more grants rejected than is good for the soul.
One grant was rejected, and 18 months later a paper that covered 'exactly' what I proposed appeared from someone in my field.
I had a pair of comments from reviewers like this:-
R1. This work has been done before by others.
R2. This work cannot be done.
I like the revised suggestion of Maurizio Morabito. There might be a problem though with “big science” projects that require large amounts of funding over many years: perhaps they should be excluded from the lottery.
The current system could be analogized with a command-and-control economy, which we know is poor.
From the linked D W Miller paper J Inf. Ethics, 2007
'Unassailable Paradigms-
Paradigms in the biomedical and climate sciences that have achieved the
status of dogma are,
A) Cholesterol and saturated fats cause coronary artery disease.
B) Mutations in genes cause cancer.
C) Human activity is causing global warming through increased CO2
emissions.
D) A virus called HIV (human immunodeficiency) causes AIDS (acquired
immune deficiency syndrome).
E) The damaging effects of toxins are dose-dependent in a linear fashion
down to zero. Even a tiny amount of a toxin, such as radiation or
cigarette smoke, will harm some people.
F) The membrane-pump theory of cell physiology is based on the concept
that cells are aqueous solutions enclosed by a cell membrane.
Scientists who question these state-sanctioned paradigms are denied grants
and silenced (Moran 1998). But valid questions nevertheless have been raised
about each of these established orthodoxies.'
Clearly the fault lies in the appointment of socio-political gatekeepers in charge of state controlled funding. And infiltration of the same into positions of influence, university departments, scientific societies, especially those of high profile, like the Royal Society, (find any mainstream grant issuing public body without an RS gatekeeper) and similarly for primary scientific publication houses and their editorial boards.
The irony is bitter. Oil companies have in fact financed financed huge technical research advances and founded many of the most respected research institutions, completely free of steered output, particularly in Earth Sciences in the USA. It is a heritage they should be justly proud of, but all too commonly a researcher has to claim oil company funding virginity to avoid the abuse of 'peers'.
Dr. K, hopefully not repeating others input.
I think your suggestions need a 4, perhaps 5 to help with deconstructing myths. I would offer that 4 should be about opportunity and capability linked to 5 which is competitive interaction. A large part of what makes evolution dynamic is this linkage between what is possible (opportunity and capability) with competition. As one thinks of the dynamics, it follows that what should be the most capable for success, usually has the least opportunity, if it is evolutionary, but it happens none the less.
These will feedback into 1, 2, and 3 as continuous improvement, but there must be in this system a capacity for retention. Not just the ones that show immediate promise, but, especially, those that revealed something new, but apparently impractical. That is because to rid oneself of an "impractical" knowledge is to claim prescience. I remember a writer in trying to explain the dynamics of evolution gave a thought experiment. What if you were "Lucy"? You are about to be the progenitor of humans and you saw a gorilla. Would you want your children to be strong, well adapted to their environment; or would you want this weak scavenger that has to change its environment and make up for its weaknesses as best it can, to be your children?
Your proposal, I fear, is to ask those who see the gorilla as strong, to see weak; and Australopithecus afarensis as the stronger. Though true eventually, it is a long road to sentience.
But I do agree with M Morabito. In fact, I think that as little as 10% in such a system would reveal its input in a few scientific generations. That would put measurable in as little as 10 years in fulfilling or showing promise. Of course, that is based on past performance of humans. I think you need to include if you did not in the previous proposal, some historical measurements of this human phenomena and its exponential nature with respect to scientific advancement.
I think that there is a weakness in that you do not mention the hierarchal result of determinism and perceived causality (familiarity of the assumed phenomena) in the energy field and current approach by humans. The achievements to date have been to increasing rely on more concentrated forms of energy because they meet our model paradigm for causality and predictability. Unless, we make your proposal work, we will be forced to either suffer the results of social disintegration that uncontrolled loss of energy services will foment (part 1, 2, 3 of your proposal), or we will have to put a "nuke on every block." Not sure if you are familiar with this politically motivated paradigm "a policeman on every block." With your background, your response to this USA political promise, I daresay, may be a bit ironic.
I would propose a block in your figure 1 for retention a "G. Retention and Evaluation" between E and B with a connection to C because we are not prescient. I am unsure where to put the ""missing link " H. Defeating Our Expectations (Lucy story)." In continuous improvement, one builds from past successes forward, in other words "I am prescient." I am proposing a new linkage that is a dual-path. We not only keep, but we expand, and expend Euros for the real, but the apparently impractical. We are not prescient.
Paradigms in the biomedical and climate sciences that have achieved the
status of dogma are,
B) Mutations in genes cause cancer.
They can.
D) A virus called HIV (human immunodeficiency) causes AIDS (acquired
immune deficiency syndrome).
It does.
E) The damaging effects of toxins are dose-dependent in a linear fashion
down to zero. Even a tiny amount of a toxin, such as radiation or
cigarette smoke, will harm some people.
Dose response curves are modeled as sigmoidal functions using a linear or logarithmic [Toxin] function. They are not treated as linear functions.
F) The membrane-pump theory of cell physiology is based on the concept
that cells are aqueous solutions enclosed by a cell membrane.
Given that this is probably the system that I have studied most during the whole of my working life, got to go with the Paradigm. Given that I knew Peter Mitchell quite well, and most of the old-boys who developed the chemiosmotic theory, and have half my publications in this area, do you think I am a fraud?
I've long argued that some fraction - say 20% - of research funds be distributed by lot. The incentive to use the money productively is the usual one of career-advancing, combined with the fact that lottery winners are excluded from the lottery for the next ten years. You'd need to be careful that the definition of who is eligible to enter the lottery didn't fall into the hands of a cabal like the Global Warmmongers, but every policy has its risks.
recognises the influence of Parkinson and maybe the ecclesiastical uncle here...hierarchies and bureaucracies are self-perpetuating. Given the choice and abi;lity, those folks will always allot money to their hangers-on.If research seems likely to lread to a commercial result, then you can be sure that the pharmaceuticals and weapons' companies will want to support them - if the public teat is available, they will hold back. genuine, blue-sky research needs to be encouraged, regardless of any commercial interest. How did we get teflon and the PC?
The 1960 edition of Clark and Stearn's Geological Evolution of North America compared the status of geosynclinal theory which was thought to explain "the origin of mountains from geosynclines," and Darwin's theory of "the origin of species through natural selection":
"The geosynclinal theory is one of the great unifying principles in geology. In many ways its role in geology is similar to that of the theory of evolution which serves to integrate the many branches of the biological sciences. The geosynclinal theory is of fundamental importance to sedimentation, petrology, geomorphology, ore deposits, structural geology, geophysics, and in fact all branches of geological science. It is a generalization concerning the genetic relationship between the trough like basinal areas of the earth's crust which accumulate great thicknesses of sediment and are called geosynclines, and major mountain ranges. Just as the doctrine of evolution is universally accepted among biologists, so also the geosynclinal origin of the major mountain systems is an established principle in geology."
Five years after the publication of the above geology textbook, geosynclinal theory was effectively dead. It was replaced by plate tectonics (which combined the hypotheses of continental drift and sea floor spreading into the theory of plate tectonics) and it became obvious to most geologists that geosynclinal theory never had possessed a testable explanatory mechanism for explaining the origin of major mountain ranges.
Who was instrumental in the DSDP Glomar Challenger oceanographic work? Lamont Docherty. Who founded Lamont Docherty? Bequest from the fortune of oilman Henry Docherty. Who published most on the follow up continental margin seismic stratigraphy and the Vail sealevel curve? Exxon.
Should have referenced my prior comment 1st 3 paras from
http://www.leaderu.com/orgs/arn/wiestr1007.htm
excellent post.
the lottery principle is fair
As it puts the establishment on an equal footing with the rest it is a no brain winwin prop
the same can be said for internal promotions (and, indeed hiring) policies in the nannystate and multinationals. Actually there is some serious scientific research on this, I hv to go find it. Someone found out that random promotions were more efficient than all the mongering in executive meetings diversity troika teams and all such lame shite.
the pareto suggestion is borne out of weakness
We cannot afford any longer the mediocre results that these cosy comittees produce
It should be clear to all they cannot be trusted.
i believe much more in a reverse pareto and would allocate 50% to a revised lottery based system and 50% to the commissariat populated with the priveleged.
Their challenge would be to cherrypick the better projects..
This is not dissimilar to beating the dart playing chimp at the stock market :)
I think the stress would be deep and after 5years it would be 80% for the lottery 20% to appeaseand chair the sons and dotties of labour union leaders and BBC coordinators.
new research proposals should NOT be reviewed , if one must review them, by peers in that part of the art.
rather , the reviewers should be eccentrics and people who have proved to be good at spotting the odd golden egg. This should be the realm of artists and vagabonds with 12 lives behind them. Professors who have been ostracised out of 10 universities because of weird ideas. that's the people who should be in the ERC commissariats.
there was this Helsinki Russian professor who found out something about levitation and magnets and was studying cigarette smoke above the magnet and then wondered if it would be the same one floor up and behold the smoke also wnet up straight there..that's the professors we are loooking for.
This story can be found in Szilard's book, "The Voice of the Dolphins". Szilard, for those who don't know, invented nuclear weapons. He wrote the famous letter to FDR that his friend Einstein signed.
Answer from the hero in Leo Szilard’s 1948 story "The Mark Gable Foundation” when asked by a wealthy entrepreneur who believes that science has progressed too quickly, what he should do to retard this progress: You could set up a foundation with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees.
...First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results.
...By going after the obvious, pretty soon science would dry out. Science would become something like a parlor game.
...There would be fashions. Those who followed the fashions would get grants. Those who wouldn’t would not.
There is another kind of justice than the justice of number……. There is a justice of newborn worlds which cannot be counted.
From the Ecclesiastical Uncle, an old retired bureaucrat in a field only remotely related to climate, with minimal qualifications and only half a mind.
OK Diogenes, I see a challenge.
Some of the bureaucracies I have been associated with are no more, abolished, in my view for the better, by politicians. Notwithstanding Parkinson’s perspicacious observation. Which points to the essential fact that bureaucracies should be, and in my experience usually have been, subordinate organizations. Politicians have the ultimate power over government bureaucracies and corporate directors the power over theirs.
Which, so far as government is concerned, probably all over the world, is a pretty grim situation.
My thoughts are;
1. that Mr Demetris’ suggestions have no chance of success. All supposedly rational people and bodies look after themselves, which includes supervising the spending of funds whether they are their own or merely those that they happen to be responsible for. It will be extremely difficult to accept that thought processes applied to the purpose can achieve no better success than chance. Making that case is Mr Demetris’ task.
(2) Mr Demetris should address his suggestion to those with the power to enforce it – the politicians or corporate boards. Boards and the like distributing funds would normally be constrained to act in the way Mr Demetris so deplores by their operating rules, and where this is not the case, diogenes’ Parkinsonian tendencies within the boards will generally give the results that Mr Demetris has experienced. Talking to them is a waste of time.
(3) the required revolution might be achieved if boards could be persuaded to focus exclusively on benefits rather than development of existing technology which I gather is now the norm. A difficult step for a load of involved technologists, no doubt.
So why not replace the membership of the boards by politicians? Their decisions would, however inexpertly, be the more directed for the public good but would essentially be random. Mr Demetris’ objectives would be achieved.
Pontifications in response!
Maurizio, Doug, Dearieme
I know that the “suggestion” is too groundbreaking. I do not expect that ERC is going to implement it soon. I do not expect that it would be ready to implement it for the 20%, or even 0.2%, of the funding. My purpose is to contribute in making the case that the current system is flawed and that better alternatives are possible.
Once (some time after the formation of ERC), I attended a talk by its then president and I asked a question how they think to deal with the problem that peer review tends to favour mediocrity. The president tried to reply honestly, but from his reply I inferred that he was not aware of the problem I was talking about.
John,
Thanks very much for your suggestions. But I do not plan to resubmit my proposal, unless the ERC radically changes its procedures, according to my proposal in its original or the above modified version :) Of course, I trust the randomness of goddess Tyche more than the determinism of the review panels.
Ecclesiastical Uncle,
I would not trust the politicians even for the prescreening phase (step 1): they would favour the proposals with abundance of clichés. Their decisions would not be random, they would be politically correct. The problem is that scientific panels behave like politicians.
DocMartyn, Pharos, Diogenes, Tutut, Paul,
Thanks for your thoughts and contributions.
Demetris
Most evolutinary change leads to a dead end.
Most scientific research leads nowhere.
On the other hand .....................................
This idea has legs.
Perhaps somebody with vision and some seed finance to spare,
could start a new foundation dedicated to revolutionary research.
50% of funds go to nutcases, 50% to proven scientists, steeped in orthodoxy.
Each would spur the other to perform exceptionally.
Your proposal has considerable merit. A few things which my (6months) research into innovation in renewables showed:
1. The success was inversely proportional to R&D spend, with wind energy receiving £50million of government money for R&D and being a complete disaster at one end and hydro, receiving no money in R&D spend being a British success at the other.
2. Successful wind energy companies, did not have research institutes. Riso, was a quality standards institute, which was later wrongly interpreted (by academics) as a research institute for wind
3. The successful strategy for development of renewable energy was "incremental improvement". The least successful (i.e. positively harmful) was the UK "spontaneous innovation", which I suppose could be likened to "punctuated evolution" ... or an attempt to completely change the product without keeping the design integrity.
4. The key to success was the number of units, with around 10,000 being the required number to come down the reliability curve (and cost curve) sufficiently to make them commercially viable.
If e.g. you wanted to produce a wave industry, I devised this plan:
1. Have a competition for a wave machine with £100 for each entry that producers wave energy for an hour.
2. Rerun the competition year upon year, upping the time till it was required to be continuous penalising heavily any failures (hence simulating real commercial conditions), and slowly increasing the required output 1W, 2W, 5W, 10W, 20,50,100, 200,500,1000, 2000, 5000,10,000, 20k, 50k 100k, 200k, but always focussing mainly on reliability.
The main aim is not what you would imagine, it is to create "engineering experience". I think a threshold is for around 100 engineers with a couple of decades of experience. In some sense, the government grant system is really just a job creation scheme ... even a training scheme, forcing engineers to get to understand the problems of putting wave machines in real conditions, getting them to survive the storms, working out the main failure modes, working solutions, ... and OK, a bit of increase in size, and even "efficiency", but the main efficiency is always how many are working at any one time (just look at any wind industrial estate ... see how many are not turning!)
"By lot" while better than any system where selection is deliberately made to support preconceived conclusions seems umnlikely to be the optimum.
I think the chances of getting the money back if misspent are slight.
However I think the use of prizes for actual achievement (in the case of climate research probably for finding errors in previous funded "research") give a strong incentive towards research which is useful being done with no incentive towards helping the "great and good" or those with preconceived conclusions.
--------------------------------
"the PI bases none of this research on current understanding and wishes us to accept that brand new thinking is required in all aspects of the proposed work. I find it difficult to accept that decades of climate change and hydrologic research have proven nothing useful "
Appears to be the generic version of Professor Jones' "why should I let people see my data, they just want to find fault with them" - a question no real scientist would ever need to ask.
AusieDan, Mike,
Glad to know you find that the idea has legs/considerable merit.
Neil,
The joint characteristic of the optimization methods I mention in the introduction (genetic, evolutionary and simulated annealing algorithms) is that they incorporate randomness in their mathematical representation. And they are able to locate the optimum (and the global one) with high probability, when all deterministic methods fail. The important thing is to accept the premise that the selection algorithm should include the element of randomness in such a way that does not favour a “preconceived conclusion”. The technical details to make the algorithm more efficient and effective can be discussed later.
Also, thanks for the comment re. Jones.
Thanks Demetris I regret I had not checked the full thing before making the "by lot" criticism and I can understand that some randomness as a factor in the selection process works.
Doc Martyn I don't know about other toxins but certainly for radiation the official theory of damage, known as the Linear No Threshold theory, is linear. It is also certainly wrong. Indeed, at least for laboratory plant and bactreial samples (the only ones where experiment is practical) the opposite theory known as hormesis, that low levels of this "toxin" are beneficial, is proven. LNT, is a theory originally invented by bureaucrats because it is easy, rather than by scientists on actual evidence. By promoting the hysteria of the anti-nuclear movement, I believe it has so far totalled more costly than the more recent catastrophic warming hysteria.
Well... I would comment on this but I suspect my own views may be a little bit biased and many people probably already know them!
What I will say is that I do not think this is isolated; Dr Pielke Sr has reported difficulties with NSF funding streams in the US, albeit slightly different in the details, here.
It is difficult to know exactly, but we might hypothesise that those known to be sceptical of the mainstream view of climate science have a higher hurdle to jump than those known to follow the IPCC line. We know this is the case for peer review, and it wouldn't surprise me if this was the case for funding also.