What is an Unusual Event?

My random thoughts on random events posted as a blog comment:

Suppose we have disjoint events \( (E_1,E_2,\dots,E_N),\) and corresponding probabilities for these events \( (p_1,p_2,\dots,p_N),\) where \( p_n=\mbox{Prob}(E_n)\) and \(\sum_{n=1}^N p_n=1.\) If a particular event \(E_k\) occurs, what would make us think this was in some sense "unusual" or perhaps "suspicious"? It's not enough that \(p_k\) be small, since for large \(N\), even a uniform distribution on the \(E_n\) will have \(p_k=1/N\) small. Nor is it enough that \(p_k\) be much less than \( \max_n p_n,\) since it is possible that all \(p_n\) are equal except for one event having many times larger yet still tiny probability. It's not enough if \(p_k\) is less than nearly all the other \(p_n\), because all the \(p_n\) could be very nearly equal.

What does seem to work in the cases I can think of is to choose some factor \(R \gt 1\), calculate \(\sum\{p_n: p_n \gt R p_k\}\), and see if this is close to \(1.\) To work this into a hypothesis test, we could reject the null hypothesis \(H_0\) if $$\sum\{p_n: p_n\gt R p_k\} \gt (1-1/R),$$ though the expression on the right-hand side is rather arbitrary. With this setup, what value should \(R\) be? Let \(x_0\) be a sample we have collected, and consider the standard normal and the \(p=.05\) rule, where \(\mbox{Prob}(|x_0|\gt 1.96)=0.05.\) Then \(R=3.71,\) since \(3.71\cdot \phi(1.96) = \phi(1.1),\) and \(\mbox{Prob}(|x_0|\gt 1.1)=1/3.71.\) If we wanted \(R=20,\) we would need to use a cutoff \( |x_0|\gt 3.135749,\) which corresponds to a very small standard \(p\)-value of \(0.001714.\)

Clearly, given any \(p\) cutoff, a.k.a \(\alpha\), we can find a corresponding factor \(R,\) and vice-versa. Since the \(p=.05\) rule is arbitrary, I don't see what difference it makes for the most common cases. Thus, \(p\)-value analysis seems generally ok to me in practice. My concern here is with its justification.