Gibbs sampling

Updating one piece at a time

Some posteriors are hard as a whole but easy one piece at a time. Gibbs sampling uses this. With two unknowns, say a mean $\mu$ and a precision $\tau$ (one over the variance), you do not need their joint distribution. Draw $\mu$ from its distribution given the current $\tau$ . Then draw $\tau$ from its distribution given the new $\mu$ . Repeat. Each draw is a conditional draw, and it is always accepted: unlike Metropolis-Hastings there is no accept-or-reject step. The pairs you collect form a chain that explores the joint posterior.

\mu\mid\tau,x\sim N\!\left(\frac{P_0m_0+n\tau\bar x}{P_0+n\tau},\ \frac{1}{P_0+n\tau}\right),\qquad \tau\mid\mu,x\sim\mathrm{Gamma}\!\left(a+\frac n2,\ b+\frac{\sum(x_i-\mu)^{2}}{2}\right)

(1)

Gibbs works best when the pieces are not tied tightly together. If two unknowns are strongly correlated, each conditional draw can only take a tiny step along the narrow ridge, so the chain zigzags and moves slowly. A two-variable example: if $x$ and $y$ are bivariate normal with correlation $\rho$ , then $x$ given $y$ is normal with mean $\rho y$ and standard deviation $\sqrt{1-\rho^{2}}$ . With $\rho=0.8$ and $y=2$ the draw of $x$ has mean $1.6$ and standard deviation $0.6$ . With $\rho=0.99$ the standard deviation drops to $0.14$ and the steps become tiny.

Both conditional steps with numbersSay

n=25

returns have mean

\bar x=0.5

, and the current

\sigma^{2}=4

, so

\tau=0.25

. With the prior

N(0,10^{2})

the prior precision is

P_0=0.01

. The data precision is

n\tau=6.25

. So

\mu

is drawn from a normal with mean

6.25\times0.5/6.26=0.499

and standard deviation

1/\sqrt{6.26}=0.400

. Next, with

a=b=1

and

\sum(x_i-\mu)^{2}=100

, the precision is drawn from

\mathrm{Gamma}(13.5,\ 51)

, whose mean is

13.5/51=0.265

, a variance of about

3.8

Gibbs sampling: draw each unknown in turn from its conditional distribution given the others.
No rejection: every conditional draw is kept, but you need each conditional to be easy to draw from.
Strong correlation between unknowns makes the chain zigzag and mix slowly.

Lab · what's graded

A conditional mean15 pts
A conditional spread10 pts
Drawing the mean15 pts
Drawing the precision15 pts
No rejection step10 pts
One sweep10 pts
Match the vocabulary10 pts
Strong correlation slows Gibbs15 pts

Updating one piece at a time

\mu\mid\tau,x\sim N\!\left(\frac{P_0m_0+n\tau\bar x}{P_0+n\tau},\ \frac{1}{P_0+n\tau}\right),\qquad \tau\mid\mu,x\sim\mathrm{Gamma}\!\left(a+\frac n2,\ b+\frac{\sum(x_i-\mu)^{2}}{2}\right)

(1)

Both conditional steps with numbersSay

n=25

returns have mean

\bar x=0.5

, and the current

\sigma^{2}=4

, so

\tau=0.25

. With the prior

N(0,10^{2})

the prior precision is

P_0=0.01

. The data precision is

n\tau=6.25

. So

\mu

is drawn from a normal with mean

6.25\times0.5/6.26=0.499

and standard deviation

1/\sqrt{6.26}=0.400

. Next, with

a=b=1

and

\sum(x_i-\mu)^{2}=100

, the precision is drawn from

\mathrm{Gamma}(13.5,\ 51)

, whose mean is

13.5/51=0.265

, a variance of about

3.8

Gibbs sampling: draw each unknown in turn from its conditional distribution given the others.
No rejection: every conditional draw is kept, but you need each conditional to be easy to draw from.
Strong correlation between unknowns makes the chain zigzag and mix slowly.

Lab · what's graded

A conditional mean15 pts
A conditional spread10 pts
Drawing the mean15 pts
Drawing the precision15 pts
No rejection step10 pts
One sweep10 pts
Match the vocabulary10 pts
Strong correlation slows Gibbs15 pts

Updating one piece at a time

Lab · what's graded

Log in to run your code.

Gibbs sampling

Updating one piece at a time

Lab · what's graded

Log in to run your code.