Don't tell them what they're really working on

In a Bloggingheads conversation between Eliezer Yudkowsky and Andrew Gelman, there's a mention of an idea due to Robin Hanson. As far as I understand it, the idea is as follows: if you want to learn what the effect of variable X on variable Y is, do not look at studies that estimate the effect of X on Y. Instead, look at all the studies that estimate the effects of many different variables A, B, C, D etc. on Y that use X as a control variable. The reasoning behind this is that whenever a researcher sets off to estimate the effect of X on Y, he or she may already have a preconceived notion of what that effect is, and that notion is likely to bias the results; whereas, since no one is invested in what the effect estimates of their control variables might show, it is those estimates that are more trustworthy.

This is a great idea--and if correct, it has tremendous practical implications. For example, whenever governments want to learn what the effect of X on Y is, they tend to commission studies that estimate the effect of X on Y. Exactly the wrong thing to do. What they should do instead is to commission a bunch of studies to estimate the effects of a whole lot of other (meaning not X) variables on Y, all of them such that X would be an obvious control variable; then pool all those studies and see if they agree about X.

In other words, researchers shouldn't be told what the real purpose of the study is.

An immediate objection to this is that lots of times governments commission those studies not to learn the truth, but instead to reinforce their own preconceived notions. This might be true of hot, publicly debatable points; I can't imagine though that there are absolutely no situations whatsoever in which a government agency actually wants to know what the truth is. I bet such situations are especially common in agencies that do work that has very high stakes but the details of which are removed from media scrutiny and ideological debate (e.g. intelligence). But then again, maybe the method I described above is in some form already employed by such agencies; I wouldn't know.

Hanson's simple idea has many more interesting implications. Of which I'll write shortly.