Buy

Books
Click images for more details

Support

 

Twitter
Recent comments
Recent posts
Currently discussing
  • Jun 22 - Mark Hodgson on
    COP 23
  • Jun 21 - Mark Hodgson on
    COP 23
Links

A few sites I've stumbled across recently....

Powered by Squarespace
« BBC R4: "Everything we know is wrong" | Main | Hiding the pause »
Thursday
Aug282014

Finding fraud in scientific papers

Judith Curry tweets a link to a fascinating report of a prototype technique for unearthing fraudulent scientific papers:

"The analysis revealed that Stapel’s fraudulent papers contained linguistic changes in science-related discourse dimensions, including more terms pertaining to methods, investigation, and certainty than his genuine papers. His writing style also matched patterns in other deceptive language, including fewer adjectives in fraudulent publications relative to genuine publications," the authors write.

Stapel tended to fortify his methods section with extra description and employ words like ‘‘profoundly,’’ ‘‘extremely,’’ and ‘‘considerably’’ to make his results sound more convincing and dramatic. At the same time, he also used fewer terms that might downplay significance, such as "less," "somewhat," and "merely."

Someone could have fun with this couldn't they!

PrintView Printer Friendly Version

Reader Comments (31)

Like leftwing folks adopting 1984 as a manual rather than as a warning, I fear that some of the motley crew thriving under the 'Climate Science' rubric will do the same with this study.

Aug 28, 2014 at 2:42 PM | Registered CommenterJohn Shade

Hmmm. So Stapel is a sample of one? What about the others. And the false positives and false negatives?

A lot of a successful academic's papers may actually be written by their students or postdocs. These will change over time, and their English may be of variable provenance and quality.

The language flags certainly strike me as that of a less than high calibre scientist, but I'm not convinced it is necessarily a reliable indicator of fraud.

Aug 28, 2014 at 3:00 PM | Unregistered Commentermichael hart

robust
rigorous
transparent
open

Adjectives that can indicate something fishy.

Aug 28, 2014 at 3:02 PM | Registered CommenterMartin A

Wol and I will be leaving the tent. We may be gone some time ...

http://thepointman.wordpress.com/2013/10/25/the-climate-wars-and-agent-deep-woolabra-wonga/

Pointman

Aug 28, 2014 at 3:29 PM | Unregistered CommenterPointman

Climate hype papers are a rich target of low hanging fruit for this sort of anaylsis. I hope climate science papers attract the scrutiny they so richly deserve.

Aug 28, 2014 at 4:03 PM | Unregistered Commenterhunter

It might be possible to detect the fraudulent for the genuine article in social psychology with words like ‘‘profoundly,’’ ‘‘extremely,’’ and ‘‘considerably’’, but in the area of climate science it is part of the standard vocabulary.

Aug 28, 2014 at 5:19 PM | Unregistered CommenterKevin Marshall

A possible approach to analysis would be an adaption of the Fog Index. Prepare a list of possible spin adjectives and weight each one in relation to its possible duplicity, misinterpretation or meaningless. Then carry out a word search on the document to identify the loading of these words as a proportion to the total word count. Prepare a list of genuine acceptable papers and carry out a similar procedure to determine a baseline. Any document registering a signficant diversion outside the base line should be considered dubious.

Aug 28, 2014 at 5:23 PM | Unregistered CommenterSpen

Stapel received the "Career Trajectory Award" from the Society of Experimental Social Psychology in 2009 - Wikipedia

Many similarities with Lewandowsky and Cook, including pseudo-scientific characterizations of unbelievers. Hope they all share the same 'career trajectory.' ;-)

Aug 28, 2014 at 5:36 PM | Unregistered CommenterJack Maloney

@ Martin A at 3.02 pm.
Add peer reviewed

Aug 28, 2014 at 5:40 PM | Unregistered CommenterBritInMontreal

Add an absence of experimentally verified facts.

Aug 28, 2014 at 5:55 PM | Unregistered Commenterturnedoutnice

Oh dear. Piltdown Mann is in deep trouble.

Aug 28, 2014 at 6:04 PM | Unregistered Commenteresmiff

Read the whole thing...

http://www.grindtv.com/outdoor/nature/post/racetrack-playa-mystery-death-valley-solved/

one more thing to add to the warmlist.

Aug 28, 2014 at 8:13 PM | Unregistered Commenterac1

1984 was about the left wing in the Soviet Union.

Aug 28, 2014 at 8:58 PM | Unregistered CommenterPathway

Bish writes: "Someone could have fun with this couldn't they!"

========

I'm usually not a betting man but I would bet this may see use in the world of 'Climatology'.

Aug 28, 2014 at 11:31 PM | Unregistered Commentereyesonu

Pointman,

No need to exit the canvas, hang around. Your peer reviewed spoofed papers will be far better constructed than the main stream and I believe you and Wol, he or she who cannot be named, are perfectly safe from the inquisition. Little do they know that you know what they know. Wink, wink, nudge, nudge, say no more, say no more.

Aug 29, 2014 at 12:02 AM | Unregistered CommenterMike Singleton

Update.

robust
rigorous
transparent
open
peer reviewed
. . . . . . . . (h/t BritInMontreal)
consistent with

Adjectives that can indicate something fishy.

Aug 29, 2014 at 12:04 AM | Registered CommenterMartin A

"Like leftwing folks adopting 1984 as a manual rather than as a warning, I fear that some of the motley crew thriving under the 'Climate Science' rubric will do the same with this study." --John Shade

Yes, if by 'rubric' you mean a largish rock.

Aug 29, 2014 at 1:25 AM | Unregistered Commenterjorgekafkazar

i think "consistent with" and robust are all you need.

Aug 29, 2014 at 3:29 AM | Registered Commenterjferguson

The use of strong adjectives reminds me of the language used by Michael Mann to characterize Steve McIntyre's objections to the hockey stick. (As cited in The Hockey Stick Illusion.)

Aug 29, 2014 at 8:47 AM | Unregistered CommenterDagfinn

"1984 was about the left wing in the Soviet Union."
Well in fact '1984', like 'Animal farm' was about totalitarianism of any hue. It is amusing that it is quoted by so many right-wingers who clearly know little about George Orwells other writings, including this; "Every line of serious work that I have written since 1936 has been written, directly or indirectly, against totаlitarianism and for democratic socialism, as I know it.". If you don't like right-wingers being called silly names then you should avoid painting all left-wingers as Stalinists!

Aug 29, 2014 at 8:57 AM | Unregistered CommenterJamesG

What about '97%' ?

Sorry, I have to spell it out, don't I: ninety seven percent.

Aug 29, 2014 at 9:26 AM | Registered CommenterRobert Christopher

"Multiple lines of evidence " is a good one

Aug 29, 2014 at 9:32 AM | Unregistered CommenterSankara

What seems to be missed by this “research” is that Stapel’s papers were peer-reviewed. To criticise Stapel’s papers, therefore, is not just to criticise Stapel, but also to throw the whole principle of the infallibility of peer-review into question. How odd that this idea seems to have been overlooked.

I do find a report like this to be profoundly suspect, extremely unsettling, and considerably untrustworthy. That less than 97% are not somewhat alarmed but are merely surprised makes the whole thing reek of fraudulence (or should that be “Freudulence”?).

(I say, this is rather jolly good fun, isn't it!)

Aug 29, 2014 at 10:48 AM | Registered CommenterRadical Rodent

Filter for finding CAGW papers:

might, may, could, possibly, probably, potentially, scenario, model, linked, associated, correlated, range, error, uncertainty, anthropogenic, man made, serious, severe, catastrophic.

Aug 29, 2014 at 11:40 AM | Unregistered CommenterNW

'Consistent with' has become the latest verbal sleight of hand.

In objective, impartial terms it actually has no meaning or value whatsoever. Last years Somerset floods were 'consistent with' climate change, or were 'consistent with' lots of rain! Take your pick.

It's weasel words, designed to fool the casual observer (or friendly hack) into confirming the catastrophic message whilst simultaneously offering not a shred of solid scientific or refutable value, thus enabling the user to cover their own arse. It can be either reiterated or distanced from at a later date as required. Very slick.

Aug 29, 2014 at 11:42 AM | Unregistered CommenterCheshirered

I thought regulars here knew full well that peer review is only a plausibility test, not any form of validation.
"... the whole principle of the infallibility of peer-review" is a fiction, unfortunately used by both sides. They use it to give equal weight to studies of differing value e.g. editorial from pressure groups versus papers in scientific journals.
Perhaps the point has been overlooked because it is trivial?

Aug 29, 2014 at 11:50 AM | Unregistered CommenterSkeptical Chymist

"... the whole principle of the infallibility of peer-review" is a fiction, unfortunately used by both sides.
True. But you try and back your argument up by referring to a paper that has NOT been peer-reviewed. There are many, on both sides of the argument, who would argue that a peer-reviewed paper has been thoroughly vetted for errors, so the paper cannot be wrong in its conclusions in the field of its discussion, no matter the views of others about its shoddiness (look at the ructions that “The Hockey Stick Illusion” caused, the after-shocks of which are still rumbling away), whereas a non-peer-reviewed paper can be summarily dismissed, without any need to view it and form your own opinions.

I suspect that the main reason the point has been overlooked is that the author does not want to draw people’s attention to the point.

Aug 29, 2014 at 9:01 PM | Registered CommenterRadical Rodent

Gergis et al's (peer reviewed) paper was shot down in flames by Jean Sibelius and Steve McIntyre (not peer reviewed, except in the real meaning of the term).

Aug 30, 2014 at 9:03 AM | Registered Commenterjohanna

Further update.

robust
rigorous
transparent
open
peer reviewed
. . . . . . . . (h/t BritInMontreal)
tipping point
consistent with . . . . . . . . .
(h/t Cheshirered)

Adjectives/words that can indicate something fishy.

Aug 30, 2014 at 7:32 PM | Registered CommenterMartin A

It looks like it was just me but terms such as 'homogenization', 'emulation' and 'normalization' had me wondering what the hell was going on. I even came across 'deinhomogenization' at GISS.

Sep 2, 2014 at 11:19 AM | Unregistered CommenterRobert B

If you applied this test - the danger is that almost all science papers might be found to be dodgy.

Sep 11, 2014 at 7:15 PM | Unregistered Commentermike Haseler

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>