Maximum likelihood

The parameters that make the data most likely

Suppose you believe the data come from a family of distributions with an unknown parameter $\theta$ (the mean and variance of a normal, the rate of a Poisson). The likelihood $L(\theta)=\prod_i f(x_i;\theta)$ is the probability, as a function of $\theta$, of the data you actually saw. Maximum likelihood estimation picks the $\hat\theta$ that makes the observed data most probable. It is the workhorse behind regression, volatility models and most of machine learning.

In practice you maximise the log-likelihood $\ell$: a product of many small numbers underflows and is awkward to differentiate, while the sum of logs is stable and has the same maximiser, because the logarithm is increasing. Setting the derivative to zero gives closed forms for the standard cases. For a Poisson sample $\hat\lambda=\bar x$. For a Bernoulli sample $\hat p$ is the proportion of successes. For a normal sample $\hat\mu=\bar x$ and $\hat\sigma^2=\frac1n\sum(x_i-\bar x)^2$, dividing by $n$ rather than $n-1$, which makes the variance estimate slightly too small in small samples.

• Likelihood: the probability of the observed data as a function of the parameter.
• MLE: the parameter that maximises it; maximise the log-likelihood for stability.
• Properties: consistent and asymptotically normal under mild conditions, but not always unbiased in small samples.

Lab · what's graded

• A Poisson rate10 pts
• A normal variance15 pts
• A win rate10 pts
• A log-likelihood15 pts
• What MLE does15 pts
• Properties of the MLE15 pts
• Why maximise the log-likelihood20 pts