Bayesian Inference Nonsense

I was reading the article on Bayes' Theorem on Wikipedia, which currently contains this bit on Bayesian inference:

Suppose we wish to know about the proportion r of voters in a large population who will vote "yes" in a referendum. Let n be the number of voters in a random sample (chosen with replacement, so that we have statistical independence) and let m be the number of voters in that random sample who will vote "yes". Suppose that we observe n = 10 voters and m = 7 say they will vote yes. From Bayes' theorem we can calculate the probability distribution function for r using

From this we see that from the prior probability density function f(r) and the likelihood function L(r) = f(m = 7r, n = 10), we can compute the posterior probability density function f(rn = 10, m = 7).

The prior probability density function f(r) summarizes what we know about the distribution of r in the absence of any observation. We provisionally assume in this case that the prior distribution of r is uniform over the interval [0, 1]. That is, f(r) = 1. If some additional background information is found, we should modify the prior accordingly. However before we have any observations, all outcomes are equally likely. [Emphasis added]
This last bit sticks in my craw, and I'll say why below. The article goes on to do some standard calculations, ending up with the conclusion:


One may be interested in the probability that more than half the voters will vote "yes". The prior probability that more than half the voters will vote "yes" is 1/2, by the symmetry of the uniform distribution. In comparison, the posterior probability that more than half the voters will vote "yes", i.e., the conditional probability given the outcome of the opinion poll – that seven of the 10 voters questioned will vote "yes" – is [.887], which is about an "89% chance".
Think for a moment about that conclusion. Is it reasonable? The experiment had 7 out of the 10 people asked saying they would vote 'yes.' Even if the total voting population is several million people, our new estimate of the probability that the referendum will pass is supposedly 89%. It doesn't seem to me that we have nearly enough evidence to reach that sort of conclusion. Mind you, I think the calculations based on Bayes' theorem are correct, and from my experience, similar calculations are common by practitioners of Bayesian inference.

But I disagree strongly with the conclusion. How is that possible? Math doesn't lie. The problem is with the assumption, which I italicized above
However before we have any observations, all outcomes are equally likely.


Even if you buy the idea that all outcomes are equally likely, perhaps justifying this assumption using a maximum entropy argument, there is the fundamental question of what an "outcome" is. I can easily choose a different assumption, no less justifiable, which results in a drastically different conclusion. It seems to me that if we really know nothing about the subset of voters who will be voting 'yes', then we might assume a prior on this 'yes'-subset as uniform on the collection of all subsets of the population. If there are N voters, then there are 2^N subsets, each of which might occur with an equal probability of 2^(-N). This uniform prior on the 'yes'-subset induces a prior on the rate r. For large N, the binomial theorem can be used to show that the prior f(r) is essentially a normal distribution with mean 0.5 and tiny standard deviation (4*N)^(-1/2). In words, practically all subsets of a large population N contain roughly N/2 elements.

Given this alternative prior, and the outcome of the survey, what is the new estimate of probability that the referendum passes? It's still essentially 50%. You could go through the exact calculation, but that's the answer you will get. With this prior, the evidence obtained from the survey of 10 people is nearly meaningless.

So who is right and who is wrong in his or her assumptions? No one can say. The fact is that this sort of blind inference is based on arbitrary assumptions, and without more information, there is literally no way to know.

2 comments:

Anonymous said...

The article's conclusion as you posted it states that:

"... the conditional probability given the outcome of the opinion poll – that seven of the 10 voters questioned will vote "yes" – is [.887], which is about an "89% chance"."

which is not the same as what you write below,

"... our new estimate of the probability that the referendum will pass is supposedly 89%."

SteveBrooklineMA said...

My post, and Wikipedia, state "In comparison, the posterior probability that more than half the voters will vote "yes"... is... 89%."

That is the same as the posterior probability that the referendum passes. Clearly the referendum passes if and only if more than half vote for it.