Model comparison and Bayes factors

Which model explains the data better?

Often you must choose between two ideas about the world. Say model $M_0$ claims a strategy wins exactly half of the days, $\theta=0.5$. Model $M_1$ allows any win rate, with a flat prior. Each model gets a score: the marginal likelihood $P(D\mid M)$, the probability it gave to the data you actually saw, averaged over its own prior. The ratio of the two scores is the Bayes factor $BF_{10}=P(D\mid M_1)/P(D\mid M_0)$. It tells how much the data favour $M_1$. Multiply it by the prior odds to get posterior odds.

Two things make Bayes factors different from simply picking the model that fits best. First, they punish complexity without any extra rule. A flexible model spreads its prior over many possible outcomes, so each one gets little probability, and it wins only if the data fit well enough to make up for that. Second, they depend on the prior, so a very wide prior on $M_1$ is punished more. As a rough scale, a factor between $1$ and $3$ is barely worth mentioning, $3$ to $10$ is moderate, $10$ to $100$ is strong, and above $100$ is decisive.

• Marginal likelihood: the probability a model gave to the data, averaged over its own prior.
• Bayes factor: ratio of two marginal likelihoods. Posterior odds equal prior odds times it.
• Built-in Occam's razor: flexible models spread their prior thin and need the data to pay for it.

Lab · what's graded

• The fixed model15 pts
• The flexible model15 pts
• The Bayes factor15 pts
• The posterior probability15 pts
• Reading a factor of 42715 pts
• Match the vocabulary10 pts
• Why complexity is punished15 pts