Momentum RMSProp and Adam

Remembering past gradients

Plain gradient steps treat every iteration on its own, so they zig-zag across narrow valleys and crawl along flat ones. Momentum keeps a velocity, a running sum of past gradients: $\mathbf v\leftarrow\beta\mathbf v+\mathbf g$, $\mathbf w\leftarrow\mathbf w-\eta\mathbf v$. Gradients that point the same way keep adding up, while gradients that flip sign cancel out. With a constant gradient the velocity grows toward $1/(1-\beta)$ times the gradient, so $\beta=0.9$ ends up taking steps ten times larger than plain descent.

RMSProp fixes a different problem, parameters on different scales. It tracks a running average $s$ of the squared gradient of each coordinate and divides the step by $\sqrt{s}$, so a coordinate with a large, noisy gradient takes small steps and a coordinate with a small gradient takes bigger ones. Adam combines both ideas: it averages the gradient ($m$, the momentum part) and the squared gradient ($v$, the RMSProp part). Both averages start at zero, which makes them too small in the first steps, so Adam divides by $1-\beta^{t}$ to correct that bias. The usual defaults are $\beta_1=0.9$, $\beta_2=0.999$ and $\eta=10^{-3}$.

• Momentum: smooths zig-zag and speeds up along consistent directions.
• RMSProp: divides each coordinate by the running root-mean-square of its gradient.
• Adam: both, plus a bias correction because the averages start at zero.

Lab · what's graded

• Momentum velocity15 pts
• The long-run amplification15 pts
• RMSProp average10 pts
• The RMSProp step scale15 pts
• Adam's first step15 pts
• Match method and idea15 pts
• Why per-coordinate scaling helps15 pts