Oxford economist Simon Wren-Lewis has a blog on the subject of macroeconomics called Mainly Macro. I chanced upon it this morning via my Twitter feed.
His latest post is about policy-based evidencemaking and who you should trust. There's plenty of good stuff in there, but plenty to take issue with too. For example, being an academic, he has an overly high opinion of academics:
I know I’ll get it in the neck for saying this, but if the analysis is done by academics you can be relatively confident that the analysis is of a reasonable quality and not overtly biased. In contrast, work commissioned from, say, an economic consultancy is less trustworthy. This follows from the incentives either group faces.
In the climate debate at least, this confidence is misplaced. We have seen time after time that many (some?) climatologists produce work of extraoardinarily poor quality, jettisoning years of accumulated knowledge of statistics and even the scientific method in favour of publishing papers that give the "right" result. Cherrypicking and data manipulation are seen all too often.
Wren-Lewis's suggestion that economic consultancies are less reliable because of their economic incentives is equally unwarranted. This is not to suggest that the incentives for those working in economic consultancies are benign. My point is that academics are working under precisely the same incentives. Funding is available for work that supports the case for catastrophic anthropogenic global warming, and not for work that challenges it. The Climategate emails reveal at least one instance of environmentalists commissioning studies from climatologists and plenty of instances of climatologists working hand in hand with green NGOs. And as I reported a few years back, the BBC's Sarah Mukherjee has alleged that most climate science is funded by greens. There are few incentives to climatologists to rock the boat and some powerful ones to prevent them - ostracisation and the cutting off of funding being just two.
I don't think there is any rule of thumb that will enable people to spot policy-based evidencemaking. You just have to throw stones at other people's work to see what happens.