Bishop Hill Finding fraud in scientific papers
Aug 28, 2014 Journals Judith Curry tweets a link to a fascinating report of a prototype technique for unearthing fraudulent scientific papers:
"The analysis revealed that Stapel’s fraudulent papers contained linguistic changes in science-related discourse dimensions, including more terms pertaining to methods, investigation, and certainty than his genuine papers. His writing style also matched patterns in other deceptive language, including fewer adjectives in fraudulent publications relative to genuine publications," the authors write.
Stapel tended to fortify his methods section with extra description and employ words like ‘‘profoundly,’’ ‘‘extremely,’’ and ‘‘considerably’’ to make his results sound more convincing and dramatic. At the same time, he also used fewer terms that might downplay significance, such as "less," "somewhat," and "merely."
Someone could have fun with this couldn't they!
Reader Comments (31)
Like leftwing folks adopting 1984 as a manual rather than as a warning, I fear that some of the motley crew thriving under the 'Climate Science' rubric will do the same with this study.
Hmmm. So Stapel is a sample of one? What about the others. And the false positives and false negatives?
A lot of a successful academic's papers may actually be written by their students or postdocs. These will change over time, and their English may be of variable provenance and quality.
The language flags certainly strike me as that of a less than high calibre scientist, but I'm not convinced it is necessarily a reliable indicator of fraud.
robust
rigorous
transparent
open
Adjectives that can indicate something fishy.
Wol and I will be leaving the tent. We may be gone some time ...
http://thepointman.wordpress.com/2013/10/25/the-climate-wars-and-agent-deep-woolabra-wonga/
Pointman
Climate hype papers are a rich target of low hanging fruit for this sort of anaylsis. I hope climate science papers attract the scrutiny they so richly deserve.
It might be possible to detect the fraudulent for the genuine article in social psychology with words like ‘‘profoundly,’’ ‘‘extremely,’’ and ‘‘considerably’’, but in the area of climate science it is part of the standard vocabulary.
A possible approach to analysis would be an adaption of the Fog Index. Prepare a list of possible spin adjectives and weight each one in relation to its possible duplicity, misinterpretation or meaningless. Then carry out a word search on the document to identify the loading of these words as a proportion to the total word count. Prepare a list of genuine acceptable papers and carry out a similar procedure to determine a baseline. Any document registering a signficant diversion outside the base line should be considered dubious.
Stapel received the "Career Trajectory Award" from the Society of Experimental Social Psychology in 2009 - Wikipedia
Many similarities with Lewandowsky and Cook, including pseudo-scientific characterizations of unbelievers. Hope they all share the same 'career trajectory.' ;-)
@ Martin A at 3.02 pm.
Add peer reviewed
Add an absence of experimentally verified facts.
Oh dear. Piltdown Mann is in deep trouble.
Read the whole thing...
http://www.grindtv.com/outdoor/nature/post/racetrack-playa-mystery-death-valley-solved/
one more thing to add to the warmlist.
1984 was about the left wing in the Soviet Union.
Bish writes: "Someone could have fun with this couldn't they!"
========
I'm usually not a betting man but I would bet this may see use in the world of 'Climatology'.
Pointman,
No need to exit the canvas, hang around. Your peer reviewed spoofed papers will be far better constructed than the main stream and I believe you and Wol, he or she who cannot be named, are perfectly safe from the inquisition. Little do they know that you know what they know. Wink, wink, nudge, nudge, say no more, say no more.
Update.
robust
rigorous
transparent
open
peer reviewed . . . . . . . . (h/t BritInMontreal)
consistent with
Adjectives that can indicate something fishy.
"Like leftwing folks adopting 1984 as a manual rather than as a warning, I fear that some of the motley crew thriving under the 'Climate Science' rubric will do the same with this study." --John Shade
Yes, if by 'rubric' you mean a largish rock.
i think "consistent with" and robust are all you need.
The use of strong adjectives reminds me of the language used by Michael Mann to characterize Steve McIntyre's objections to the hockey stick. (As cited in The Hockey Stick Illusion.)
"1984 was about the left wing in the Soviet Union."
Well in fact '1984', like 'Animal farm' was about totalitarianism of any hue. It is amusing that it is quoted by so many right-wingers who clearly know little about George Orwells other writings, including this; "Every line of serious work that I have written since 1936 has been written, directly or indirectly, against totаlitarianism and for democratic socialism, as I know it.". If you don't like right-wingers being called silly names then you should avoid painting all left-wingers as Stalinists!
What about '97%' ?
Sorry, I have to spell it out, don't I: ninety seven percent.
"Multiple lines of evidence " is a good one
What seems to be missed by this “research” is that Stapel’s papers were peer-reviewed. To criticise Stapel’s papers, therefore, is not just to criticise Stapel, but also to throw the whole principle of the infallibility of peer-review into question. How odd that this idea seems to have been overlooked.
I do find a report like this to be profoundly suspect, extremely unsettling, and considerably untrustworthy. That less than 97% are not somewhat alarmed but are merely surprised makes the whole thing reek of fraudulence (or should that be “Freudulence”?).
(I say, this is rather jolly good fun, isn't it!)
Filter for finding CAGW papers:
might, may, could, possibly, probably, potentially, scenario, model, linked, associated, correlated, range, error, uncertainty, anthropogenic, man made, serious, severe, catastrophic.
'Consistent with' has become the latest verbal sleight of hand.
In objective, impartial terms it actually has no meaning or value whatsoever. Last years Somerset floods were 'consistent with' climate change, or were 'consistent with' lots of rain! Take your pick.
It's weasel words, designed to fool the casual observer (or friendly hack) into confirming the catastrophic message whilst simultaneously offering not a shred of solid scientific or refutable value, thus enabling the user to cover their own arse. It can be either reiterated or distanced from at a later date as required. Very slick.
I thought regulars here knew full well that peer review is only a plausibility test, not any form of validation.
"... the whole principle of the infallibility of peer-review" is a fiction, unfortunately used by both sides. They use it to give equal weight to studies of differing value e.g. editorial from pressure groups versus papers in scientific journals.
Perhaps the point has been overlooked because it is trivial?
I suspect that the main reason the point has been overlooked is that the author does not want to draw people’s attention to the point.
Gergis et al's (peer reviewed) paper was shot down in flames by Jean Sibelius and Steve McIntyre (not peer reviewed, except in the real meaning of the term).
Further update.
robust
rigorous
transparent
open
peer reviewed . . . . . . . . (h/t BritInMontreal)
tipping point
consistent with . . . . . . . . . (h/t Cheshirered)
Adjectives/words that can indicate something fishy.
It looks like it was just me but terms such as 'homogenization', 'emulation' and 'normalization' had me wondering what the hell was going on. I even came across 'deinhomogenization' at GISS.
If you applied this test - the danger is that almost all science papers might be found to be dodgy.