An introduction to Bayesian inference

Maximum-likelihood estimates have a lot of nice features, but likelihood is a slightly backwards look at the world. The likelihood of the data is the probability of the data, $x$, given parameters that we don't know, $\phi$, i.e, $\mbox{P}(x\vert\phi)$. It seems a lot more natural to think about the probability that the unknown parameter takes on some value, given the data, i.e., $\mbox{P}(\phi\vert x)$. Surprisingly, these two quantities are closely related. Bayes' Theorem tells us that

$$\mbox{P}(\phi\vert x) = \frac{\mbox{P}(x\vert\phi)\mbox{P}(\phi)}{\mbox{P}(x)} \quad .$$ (6)

We refer to $\mbox{P}(\phi\vert x)$ as the posterior distribution of $\phi$, i.e., the probability that $\phi$ takes on a particular value given the data we've observed, and to $\mbox{P}(\phi)$ as the prior distribution of $\phi$, i.e., the probability that $\phi$ takes on a particular value before we've looked at any data. Notice how the relationship in (6) mimics the logic we use to learn about the world in everyday life. We start with some prior beliefs, $\mbox{P}(\phi)$, and modify them on the basis of data or experience, $\mbox{P}(x\vert\phi)$, to reach a conclusion, $\mbox{P}(\phi\vert x)$. That's the underlying logic of Bayesian inference.¹⁴