Tuesday, January 24, 2012

Majority, average, what's the difference

So, I was just listening to this here podcast. It started off with a discussion of James Surowiecki's The Wisdom of Crowds which talks about the fact that when you're trying to estimate some quantity, an average over a large number of individual guesses of people picked at random will be closer to the truth than an expert's opinion. The interviewed guest gives an example of Francis Galton's observation that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox's true butchered weight than the separate estimates of any of the cattle experts). He then goes on to say that the reason behind this and similar, seemingly magical phenomena, is Condorcet's Jury Theorem: Take a group of people each of whom is more likely to get the right answer than the wrong answer and ask them the question. As size of group increases, the probability that the majority gets the right answer approaches 1 in the limit; the same holds for pluralities. It is also why surveys are accurate.

OK, something's not quite right here. Lots, actually. In the context of the ox example, what does it mean that each group member is "more likely to get the right answer than the wrong answer?" Weight is a continuous variable, so for each group member, the probability that they'll guess the right answer is precisely zero. Grad school was a long time ago, but I seem to remember something about Condorcet's Jury Theorem being applicable only to situations of binary choice (hence the word "Jury" in the name). The average of individual guesses of a continuous quantity, and a majority pick from two alternatives, are very different things. Also, the reason surveys work is Central Limit Theorem and not Condorcet's Jury Theorem. The only thing those have in common is the word "Theorem" in the name.

No comments:

Post a Comment