Bishop Hill The peer review game
Feb 19, 2014 Journals There is an interesting letter in Nature this week. In-Uck Park of the University of Bristol and his colleagues have adopted something of a game-theoretic approach to try to understand aspects of the peer review process.
The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists’ decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers’ behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.
Which is a pretty interesting result, and one which I think will ring true with many readers at BH at least. Here's an excerpt from the conclusions:
Science may ...not be as self-correcting as is commonly assumed, and peer-review models which encourage objectivity over subjectivity may reduce the ability of science to selfcorrect. Although herding among agents is well understood in cases where the incentives directly reward acting in accord with the crowd (for example, financial markets), it is instructive to see that it can occur when agents (that is, scientists) are motivated by the pursuit of truth, and when gatekeepers (that is, reviewers and editors) exist with the same motivation. In such cases, it is important that individuals put weight on their private signals, in order to be able to escape from herding. Behavioural economic experiments indicate that prediction markets, which aggregate private signals acrossmarket participants, might provide information advantages.Knowledge in scientific research is often highly diffuse, across individuals and groups, and publishing and peer-review models should attempt to capture this.We have discussed the importance of allowing reviewers to express subjective opinions in their recommendations, but other approaches, such as the use of post-publication peer review, may achieve the same end.
Reader Comments (100)
The classical case of peer review failure is 1981_Hansen_etal.pdf . This primary claim of 33 K terrestrial GHE assumed a -18 deg C zone in the upper atmosphere in radiative equilibrium with Space. There is no such zone. Had the reviewer(s) done their job, they would have prevented ~3x exaggeration of the real GHE for the flat Earth model, and much wasted investment!
'raises the possibility that, under certain conditions, science may not be self-correcting.'
No history tell us that science has, in the all important consensus approach, has 'failed' to be self-correcting. a number of times. Its the work outsiders who have often been initial attacked , ridiculed and ignored before then prove to be right , that have lead to correction .
Peer review has lots of dirty aspects than science would rather the public did not know about, for if it these would be wider know scientists can pretty much forget its 'iconic' status and then their status with society .
"converging on an incorrect answer"
I think the Bishop has been saying that for some time.
Once more, I shall repeat myself, and raise the spectre of Diederik Stapel for those devout believers in the infallibility of “peer-review” (and I suspect it is a fame he would far rather do without). All you need to do is read the caption under the photo: “…published fabricated data in 30 peer-reviewed papers.”, to realise that peer-reviewed papers can be completely and utterly WRONG. Knowing that, one has to wonder about the thinking behind the reviewing “peers”.
Climate Psience may ...not be as self-correcting as is commonly assumed; herding and "converging on an incorrect answer"
Couldn't put it better myself :-)
Eeps! Continue to forget to sign in…
Note: it is 30 – THIRTY – 3-0… not 1 (one) that slipped through the net, but 30!
If that can occur with one “academic’s” papers, then it is entirely conceivable that it could happen with others. Indeed, there was a sequence of e-mails in the “Climategate” set where the protagonists discussed the destruction of a peer-reviewer’s (Chris de Freitas, of the University of Auckland) career because he criticised one of their papers that he was reviewing.
Shame that he fell into this trap:
Obviously, he believes that "scientists" are purely motivated by seeking truth, when it is obvious that many are more motivated in getting funding.Seems Kahan has mostly projected
In these circumstances I always like to remind people of the Ediacaran story, where scientists refused to self-correct for decades.
The story includes the usual Nature rejection of the work of a scientist followed by the usual Nature approval of the same work, done by a Scientific Authority.
If we are using a game analogy, that fits neatly with idea of 'victory conditions'. i.e. a game (which is a model) has a set of fixed rules, you must use these rules to beat your opponent, but how you beat him has been prejudged (i.e. checkmate the king)
Peer review is very much like this.
Of course in the real world, Alexander the great used tin soldiers to practice and get an idea of how a battle might go, but he never forgot that in the real world there are no rules. Peer review is similar, quite useful to get you started and to teach a student the basics, but useless where it really matters.
In such cases, it is important that individuals put weight on their private signals, in order to be able to escape from herding.
This begins to sound dangeroulsy like having a 'feel' for things.
There are four basic forms of peer review. 1) rigorous deconstruction, 2), replication, 3) quick review, 4) prejudice review.
Rigorous deconstruction is what the media and the misled public believe is the norm in the peer review process. Actually it is the rarest of all, it would seem especially so in main stream climate science. It is the most effective way to properly progress science.
Replication is similar but tests methodology rather than conclusions. Also often very effective but again apparently rare in climate science.
Both these methods are, of course, very time consuming for the reviewer.
Quick review is seemingly the generally adopted standard and usually consists of a cursory overlook to expose any glaring errors but does not really examine anything. Quick and easy for the reviewer.
Prejudice review, seemingly the default in climate science. Content does not matter, if it agrees with your own prejudice/work it is good, if it doesn't it's bad. This is particularly likely to occur where ego or political ideology dominate. An interesting example of this goes all the way back to Rutherford, Father of Nuclear Physics, where many believe he held back further understanding of the science for decades by off-handedly dismissing the ideas coming from the next generation of researchers.
I don't know if it's covered in the full article, but Millikan's oil drop experiment is the classical example of 'herding'. See under Feynman here
https://en.wikipedia.org/wiki/Oil_drop_experiment
Plus ca change
There's another paper in this weeks Nature: 'Confirmation of Ursidae defecatory behaviour in Forest environments.'
Peer review is just one form of quality control and, like all others, it is sometimes useful but never perfect. When I was a very lowly researcher, my supervisor would occasionally toss me a manuscript with the request "tell me what I think of this". I like to think my reviews were useful, but the scientific community was not necessarily getting what it thought.
Peer review made some kind of sense when snail mail was the fastest form of communication available to ordinary people. Whole world review via the world wide web is much more likely to detect error.
Solent
Thanks for the Feynman link. As he put it:
It's a thing that scientists are ashamed of—this history—because it's apparent that people did things like this: When they got a number that was too high above Millikan's, they thought something must be wrong—and they would look for and find a reason why something might be wrong. When they got a number close to Millikan's value they didn't look so hard. And so they eliminated the numbers that were too far off, and did other things like that...
It looks like they still are doing 'things like that'...
Radical Rodent Feb 19, 2014 at 9:34 AM
" Knowing that, one has to wonder about the thinking behind the reviewing “peers”."
The 'peer' bit is the reviewer having enough knowledge to detect a possible lack of logic, clarity or fact so, as an example, some knowledge of Wave Mechanics and its practical applications would be useful when reviewing papers in that area. If I read a reputable review of Shakespeare's All's Well That Ends Well, I would expect it to have been written by someone who had perhaps seen a few of his other plays and had taken an interest in them!
This is common sense, but what is not is believing that peer reviewed papers are correct, THE TRUTH, the facts are settled!
Peer review is a human activity. When done properly, it is a process that attempts to squeeze errors, mistakes, lack of clarity and confusion from statements that are trying to advance our knowledge, with little consequence for much else! (See where the problem lies!)
It should be a process of gradual escalation of possible embarrassment:
1) self review - to make it presentable to others
2) review by friendly colleagues, to eliminate dreadful gaffs, spelling mistakes and school boy errors in grammar
3) review by not-so-friendly colleagues, to eliminate subtle gaffs
4) review by non-colleagues. It is now beginning to get serious!
5) official peer review or out on the Web
Each stage is there to stop mistakes reaching the next stage - minimising embarrassment !
It is done in most walks of life: the Law, Engineering, Medicine on a day to day basis. It is just that, in Science, the number of knowledgeable people in a specialised field is often very small and group-think can take over, as can common interests.
What is needed is an early stage that at least offers suggestions for what outside skills should be introduced to improve quality. It would also discourage group-think!
I would suggest an independent statistician, an experienced graph designer (putting in proper axes, using colours and labels that cause least problems for the colour blind of us and the most appropriate scales). Signal processing (spectral analysis is best done on the initial data, not merged/smeared data) and data storage (getting standard formats would save a lot of resource) would be quite common suggestions. I am sure there would be other areas that could be suggested.
Allan M
There is absolutely nothing wrong with "having a feel for things". Perhaps if some of our climate experts had had a a "feel" or they had followed their instincts we might not be in the bind we are in.
My interest in pursuing the global warming argument began with a sense that "this doesn't smell right" and I imagine there are quite a few others here that are the same.
The danger is that you let that "feel" override the facts and the data but without that feel or instinct or hunch in the first place a lot of good science would never have got done.
@ Omnologos.
Apart from a few Greeks in around 300-400 BC, the general consensus was that the Earth was at the centre of the known Universe, & that the Earth was flat, & that if you sailed too far you would fall off the edge, or be eaten by sea-monsters. This consensus was maintained for over a thousand years, reinforced most strictly under torture & pain of death by the then UNIPCC of its day, the Holy Roman Catholic Church. It wasn't until the likes of Copernicus & then Gallileo demonstrated by observation that this could not be the case, but still it was some time before openness could be achieved. In the 19th Century, the general consensus about Malaria was that it was an airborne disease, hence it's name, "bad air". It was several years before it was recognised that it was transmitted by mosquitos, as the bacterial world was slowly uncovered by Victorian scientists. In late Victorian England, the President of the Royal Society proclaimed that "heavier than air flying machines were impossible!". We now know different. In the 1970s, the general consensus of stomach ulcers was that they were the result of stress & eating too much spicy food, & the two antipodian scientists who thought otherwise were all but vilified by their peers, but they pursued their ideas until eventually they were recognised, winning a Nobel prize for their research. However, with Global Warming, the world faces something slightly different, particularly with Agenda 21 & the EU, when one gets the likes of Connie Heddergaard saying things like "but even if Global Warming turns out to be untrue, isn't it still a good thing to do what we are doing by using less of everything?" It is predetermined at this point in time that the course of action being taken is the correct one. SO I guess my point is that scientists are loading the peer review in any case to fulfil an agenda, whether it b the 21 variety of another!
Solent - apparently Robert Millikan was rather full of himself. So much so our physics prof said that his colleages proposed defining the standard unit of modesty as the "kan".
Alan the Brit
Before Copernicus, John of Holywood, who died c. 1236, depicted the world as round in his work De sphaera mundi which was written for use at the University of Paris and taken from earlier Arab sources. It also included an estimate of the earth's circumference.
I don't know why people think scientists should be some kind of breed apart. A few want fame, fortune or notoriety, but most just want a decent living, secure job, pleasant work place and a happy family. You do not get that by being the black sheep. Do what the Government want as they dish out most of the funding. Do what your employer wants or lose your place on the money tree. Don't irritate your colleagues by heavily criticizing them. Don't rock the funding boat because your colleagues won't thank you for that. Don't tell traceable lies but a few omissions or inaccuracies won't matter very much. Waffle a bit if pressed and don't step out of line. This is how 99.9% of people behave. Why should "scientists" be different?
All all true scientist know, science exist to consolidate the consensus or the correct view.
As motto says "Scientia probatur quod aer sit infinita ex phlogiston"
There would not be so much going on about peer review if people would grasp that 'passing peer-review' does not in the great majority of cases mean that a paper is being given some endorsement, approved as correct/right/good.
The root of the problem is that warmist scientists have quite deliberately, knowingly, misled the uninformed by claiming just the opposite. The problem is not with peer review, which works adequately within its own constraints, the problem is with those who mendaciously claim for it qualities it does not have, to serve their own interests.
@Ivor Ward: the reason why scientists should be different is because if they are not 'different' they are unprofessional.
It used to be the case that such a charge would render you unemployable. I hope this once more becomes the case because otherwise we face the end of the Enlightenment. This has allowed 200+ years during which self-regulated freedom of thought and speech became the norm. Without it we face violent, totalitarian oppression. It is that stark.
How about the herding that comes from having a common paymaster who selects and rewards people and projects ?
The paper ignores humans "Ego". Most scientists only have 4-5 truly original ideas which they have conceived by the age of 30. They say most scientists most fruitful work ranges from the last year of the undergraduate degree, through the doctorate and then the first post doc, perhaps only 8 years in total. Scientists such as Newton,Clerk Maxwell, Einstein and Darwin may have conceived most of their ideas by the age of 30 but they then had time to reflect on them before they were published. Newton did much of his work while on the family farm from from 1665-1667 , at the age of 22-24 but did not publish "Principia " until 1687 at the age of 44.
Clerk Maxwell productive career was completed by the time he was 40 and Einstein by the time he was 31-32. I would suggest that it is unreasonable for many scientists to make significant contributions after the age of 45 and instead of trying to be original they should be mentoring the younger generation and running departments.
In the modern age of publish or perish , I think considerable resources are wasted undertaking inconsequential research which is often being undertaken at several universities. In the area of environmental research samples need to be taken over 10s, 100s, 100s and 1,000,000s of sq kilometres and over periods of ranging from hours to 1000s of years and then analysed by teams of scientists including those with high levels of skill in statistics.
What we have are far too many papers which because the research money is spread too thinly produce results which are uncertain and therefore easily manipulated for political reasons.
When it comes to turning off the pumps and stopping dredging, The EA should have had physical models constructed at Teddington to assess what would have happened .
http://www.telegraph.co.uk/topics/weather/10644101/How-Somerset-Levels-river-flooded-after-it-was-not-dredged-for-decades.html
If one looks at the " The Dambusters " Barnes Wallis undertakes extensive testing using scale models at Teddington.
So what we have are far too many people who either have no original thoughts or already had them , have chasing too little money , undertaking research which produces uncertain results .
What we need are 1 in 10 scale models of rivers such as the Parrett and river catchments to undertake physical modelling. Most of the physical geography /environmental science degrees should be shut down and mandatory A Levels in Maths and Chemistry should be required for such degrees. Many of the people with degrees in physical geography or environmental science have less scientific ability than some one with a HNC in Physics , Civil Engineering or Chemistry taken 30 years ago and a lot less useful.
Physical models , built at 1 in 10 scale of all high risk flood plain should be constructed and tests run by civil engineers, agricultural engineers, statisticians ,hydrologists and geologists to assess the situation. We need to return to the pre mid 60s when there was clear distinction between top universities and research institutes who undertook research and minor ones and polytechnics whose primary job was to train competent scientist and technicians at the ONC,HNC, degree and master level. Minor universities and polys would be judged on not whether academics had published papers or undertook research but how competent were their graduates and how useful they were to local companies.A Sampson in " Changing Anatomy of Britain 1982" says that the assessment of published papers led to the decline of the technical universities such as Salford and Aston . These universities were never meant to produce publishable research , their's was to train competent engineers and scientist and undertake research for local companies who did not want it published because of commercial reasons.
If the EA had competent Civil Engineering Technicians trained to HNC level who had installed extensive monitoring equipment from 1995 onwards ( founding of EA ), by 2005 they would have had 10 years of data upon which to base decisions.
bill
Good point well made. Climate science reporting has twisted the concept of publishing an idea or theory, which most of climate science is, into some sort of validation of that idea.
Or in more general terms they use peer review as if it were Facebook Likes. Taking something that is actually superficial to mean quality or correctness.
Robert Cialdini would be fascinated.
Alan the Brit,
No-one seems to have seriously thought the world was flat (note that there is a huge hint as to the shape in a clear night sky, which is the moon, and a more subtle one in daylight in the form of the sun). The church inherited the Greek learning on the shape and size of the world, and was responsible for passing this on (most Greek learning is in medieval manuscripts). No pope (or orthodox patriach) has ever claimed the world was flat - possibly because all had travelled beyond the horizon at some point (at which point you work out the world is not flat pretty quickly).
The issue which caused Gallileo and others problems was heliocentrism, the much more difficult to prove idea that everything revolved around the sun, not the earth (try making this case when the body you are on appears stable, you have no secondary reference point, and the only other easily observable celestial body does in fact demonstratably revolve around the earth). You can kind of understand why people found this difficult to accept.
And there is an issue of comparing modern science, however politically motivated, with problems with the church experienced by earlier 'scientists', which is that the church was relying on theology to oppose arguments, not science. The Hockey Team and their ilk have to clothe their agenda in science - only a few true believers will rely on appeals to Gaia or whatever - so although they can pervert the debate, they still have to debate the science. The church could guillotine debate by reference to theology; the nearest we have is the clearly unsuccessful (and not accepted even by most of the scientists pushing the man-made global warming narrative) idea of a consensus.
All in all, I wouldn't use the flat earth as a comparision, as all it does is irritate people by repeating silly stories (I suspect the Victorians here) that were never true. Use something like the refusal to accept the existence of continental drift instead, where a clear consensus was challenged and shown to be wrong.
1981_Hansen_etal.pdf .
Feb 19, 2014 at 9:22 AM | Mydogsgotnonose
MDGNN - do you have a working link (or the exact title)? Thnx
@ Messenger,
Thank you. I had never heard of him. Isn't the "free & open" exchange of information a wonderful thing. You could have been like a Climatologist & said "I know something but I am not going to tell you how I worked it out!" ;-) Will look into John of Holywood, thanks again.
Bill: "There would not be so much going on about peer review if people would grasp that 'passing peer-review' does not in the great majority of cases mean that a paper is being given some endorsement, approved as correct/...:
You know what Bill I have just had an exchange with Rog Tallbloke on Twitter re this very topic. My point and Rog concurred was that "peer review" was only stage one and that the ideas in the paper would have to pass muster with the scientific community before they could be taken seriously. However, whether it's the number of papers they have to read given the output of papers, or some other reason any published paper, particularly in climate science and particularly if it supports the consensus is accepted at the point of publication, by cliscis as gospel truth. So they do seem to believe tha peer review
Nerd and Rookie, denizens of the SkS blog , not cliscis themselves, can produce a paper using computer models that proves the Arctic is 6C higher than thought and it is instantly bruited about the clisci community as an important breakthrough. Or you could look at the reception of MBH1998 and the M&M 2003 rebuttal and how scorn has been, and still is, poured on M&M by the sticking community.
That would be "clisci community"
@Martin A: interesting, GISS has blocked access to it via Google**, presumably because of the major errors in the paper for which Hansen as lead author was responsible. Damage limitation? It is still there as: http://pubs.giss.nasa.gov/docs/1981/1981_Hansen_etal_1.pdf
The correct GHE is obtained from the radiative equilibrium temperature for 341 W/m^2. It is 4 to 5 deg C making ~11 K GHE. The 3x positive feedback con came from using 238.5 W/m^2 SW thermalisation which gives -18 deg C, 33 K GHE.
It's a neat coincidence that this 'mistake' has lasted 33 years!
**There's a blocking script.
PS: Hansen's claim that Lapse Rate temperature difference from the surface to the -18 deg C zone is the GHE, from Sagan via Houghton, would work if there were such a zone. As the -18 deg C is a flux weighted average of three processes, the heat generation and transfer in the models is wrong and you easily show CO2-AGW to be near zero.
Watchmen.
Good point. The wealth of the Church increased dramatically from idea which does not originate from the Church automatically reduces their authority and hence ability to obtain money.
Any scientists who challenge AGW threaten the ability of those who support it to obtain money . Unless one is like Darwin who had money because he married a Wedgewood , Einstein who was patent clerk , Newton a Fellow of Trinity and land owner one is dependent upon the state. The welfare aspects of the Church have been replaced by the State.
Status enables a scientist to become a Professor which in the UK comes with a salary of about £75K which with another say £10k from outside work comes to £85K. This is well within the top 10% earners in the UK and outside London and SE can provide a comfortable upper middle class lifestyle.
Unpacking the potentially cryptic omnologos comment "Seems Kahan has mostly projected" (it took me a while),
Kahan says that the public form their view of scientific issues based on the thinking of the herd they belong to. This paper says academics have a herd instinct. So Maurizio says Kahan is projecting the academic way of thinking onto the public.
Thank you Paul - yes, that's what I meant ;)
I can now understand why Kahan is so fond of his findings, even if his r-squared is very low for this scientist-engineer's eyes.
A scientist who doesn't know the history of science including its innumerable disasters is not a scientist, but just another unwitting soldier in the state-sponsored quest for potentially disastrous half-knowledge.
This past 40 years has been a rerun of the Phlogiston debate. In the present case, the fake science has been to claim that the Earth's surface which has a mean Radiation Field of ~390 W/m^2, emits real net IR energy at that rate. It doesn't..
OK I analysed it
"The objective of science is to advance knowledge" .. tick
..the rest of his/her words are mostly BS
his conclusion bit
"science..self correct it's "... Science will self correct as things either work or they don't , it's just about the speed we get to truth
..ythe rest also seems mostly BS IMHO
I don't think he knows what he is talking about
- Yep Yesterday as usual a warmist tried to tell me "peer reviewed" =True
(AFAIK it only means the process has been checked by your mates in the field)
50% doesn't stand time, MMR, Korean Dog Cloning and now today signs of photoshopping in the recent Japanese stemcell breakthrough
....which was published where ? go on guess ..
Re: Millikan
While Feynman may be correct that accurate determination of the electron charge did not go smoothly after the Millikan experiment, he is clearly wrong in attributing Millikan's error to poor estimates of air viscosity: the experiment was originally done on oil droplets statically suspended in the chamber so air viscosity was not a factor (this is pretty obvious since a droplet would have to stay still long enough to have its radius measured). Interestingly, Millikan's graduate student Harvey Fletcher came up with the idea of using oil (they were using water but it evaporated too quickly) and I personally have heard him tell of going to the corner drugstore to get it. It was Fletcher who actually performed the experiment successfully the first time.
Without seeing the complete [pay-walled] article, it is difficult to see what the new perspectives are that they are claiming.
The cynic will comment that they have 'modeled' peer-review. I wonder if they applied the models to their own paper?
"Peer review" is one of those hitherto specialist pieces of academic jargon that Greenie activists have seized on in the late internet age.
I've known Greenies for at least 40 years and I swear I never heard one of them use the term "peer review" until sometime in the last five or so years, then suddenly they're all over it like they invented it.
It's no magic bullet, it's just a stage of quality control you go through and it can be as minimal as a few of your colleagues going through it or a wider industry body or some international committee that knows SFA about the matter in hand.
"Just cos it was in Nature, doesn't mean it's wrong"
very recent AMAZING Japanese stemcell #STAP breakthrough (seemed too good to be true)
Seems some of the photos were photoshopped Written up well in Chinese Science blog
... doesn't disprove study
- Still no fully independent replication of the Japanese #STAP stemcell breakthrough
Jointblog all other researchers
- Expert Opinion doubts
..also in AAAS with open comments
Geronimo, one aspect that makes clisci different from most science is that it is surrounded by amateurs (the NGO/activist tendency) who don't understand the proper meaning and role of peer review and, as I said, misled by a knowing leadership, are happy to believe/echo the idea that 'passed peer review = good'. Actually peer review is a very low threshold (in most disciplines), it is review, not re-doing or even marking the authors work, and if a work gets hammered after publication as being balls, neither the reviewer, nor the editor, nor the publisher would think themselves at fault. The work is the AUTHOR's work, for which he is responsible.
Additionally the editorial tendency is to be inclusive, because of the way science works; which is another reason why the peer review threshold is generally low. Most poor papers don't get hammered, they get ignored. In fact, most papers get ignored. So you could say there was no point publishing them in the first place. But at the same time, everyone in 'science' knows, what appears inconsequential today could be seminal in 30 40 50 years time. Knowledge is incremental, and trying to guess which increment will matter in the future is not part of most editorial mindsets.
- "primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. BS
Peer review acts as the gatekeeper to these mechanisms". BS.. Gatekeepers not needed as they HAVE KEPT a lot of good science out
- "Given the increasing concern surrounding the reproducibility of much published research" BS problems are not new
- "it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists’ decisions." ??
blah blah .."herding" ..why doesn't he say "Grouthink" for becoming part of crowd ? .... Publication bias ? & Grant Bias
.. I've heard that there is no need for peer review at all, just publish everything on the internet and let the market sort it out. It's not like the old days when with a limited publication soace bunk ideas could crowd the good ones out.
I'm afraid that the first thing that I look at these days when reading about research is to look at who sponsored/funded the research and where the researcher is based.
Regrettably, these days, this information often gives one a good clue as to what one might expect the findings to be before one has even read the report.
That's always been the case - doesn't invalidate the paper, but you do have to bear the bias in mind, and think particularly about what the writers omitted as well as included.
Unfortunately the internet commentariat is recently infected with the notion that Peer Review = Approved by GOD.
GlennD: "he [Feynman] is clearly wrong in attributing Millikan's error to poor estimates of air viscosity: the experiment was originally done on oil droplets statically suspended in the chamber so air viscosity was not a factor (this is pretty obvious since a droplet would have to stay still long enough to have its radius measured)."
A memoir by Fletcher is here. The note at the bottom of page 45 mentions that viscosity is a factor. The Wiki entry describes how viscosity entered into the calculation.
[Ed. on reading further in the Physics Today article, it becomes apparent that the Wiki entry is not entirely accurate in assuming Stokes' Law was used; the oil drop experiment appears to have produced a correction to it. So don't take the Wiki article as gospel. Which probably could be said of everything in Wikipedia.]
Feb 19, 2014 at 12:58 PM Mydogsgotnonose
Many thanks for the link.
As I've said before, please write your stuff up so we can read it in detail. Cryptic comments are only meaningful to someone who already shares your mindset.