Department of self-reference department

Previously I blogged about Robin Hanson's idea of not trusting results of direct studies. More precisely, the idea is that whenever you want to learn what the correlation is between some variables Y and X, you shouldn't look at studies in which X is the main variable of interest (let's call these direct studies) as they're likely to be biased, but instead at studies in which X is a control variable (call these indirect studies). I'll call this the Hanson Hypothesis:

Results of direct studies are biased, whereas results of indirect studies are not.

Now think about this: what if we wanted do an empirical study of the Hanson Hypothesis? That is, we want to find out the effect of variable X (whether a given study is direct or indirect) on Y (the quality of study's results). Can we do that? We can't, because we'd be treating X as a variable of interest and therefore conducting a direct study, and results of direct studies are biased (by Hanson Hypothesis).

So far so good. Now let's make it a bit more complicated by stating what I'll call the Precise Hanson Hypothesis:

Most results of direct studies are biased in the direction of researcher's prior belief about those results.

That is, if a researcher believes that, say, obesity is bad for health and then conducts a study on the effects of obesity on health, his results will show that obesity is bad for health even if in fact it's not. Now think about testing the Precise Hanson Hypothesis. Let's say you commission testing it to some researcher and he gives you his results. Can his results be informative? Well, that depends on his prior beliefs. If before conducting his study he believed that the Precise Hypothesis was false, his results will be of no use to you. If the hypothesis is false, he'll come back with results that say it's false. But, if the hypothesis is true, he'll come back with the same results (because the results will be biased in the direction of his prior belief, which is that the hypothesis is false). However, if you ask someone whose prior belief is that the Precise Hanson Hypothesis is true (e.g. Robin Hanson), his results will be informative. If he comes back with a negative result, you'll know the hypothesis is actually false (for, if it were true, someone with a prior belief that it's true could only evaluate it in the positive). If he comes back with a positive result, you'll know the hypothesis is actually true (for, if it were false, anyone would come back with a negative result, regardless of their prior belief).

Let's make this weirder still. Now think about the Strong Hanson Hypothesis:

All results of direct studies are biased in the direction of researcher's prior belief about those results.

Can you test it empirically?

It turns out that you don't have to; the Strong Hypothesis is false, and it can be shown without any empirics. For suppose it's true, and suppose also that you ask someone who believes it's true to do an empirical study of it. He does the study, and his results are that the Strong Hypothesis is true. The question is: are the results biased or not? (By biased I mean that the researcher would return results confirming his prior belief regardless of whether those beliefs were true.) Well, it's a direct study, so by the Strong Hypothesis it must be biased. This means that if the Strong Hypothesis were actually false, the researcher would still come back with results saying it's true. But if the hypothesis is false, it's impossible for anyone to have results saying it's true, so the study we commissioned can't be biased. In other words, it's an unbiased direct study, exactly what the Strong Hypothesis says can't exist. So the Strong Hypothesis is false. But we've just assumed it was true, so we have a contradiction, which means it actually is false. That's good news, I think.