How models learn

Gradient descent: always downhill

A ball on a wavy loss surface, moved by one rule: step against the gradient. Eight short chapters on what the gradient is, how big a step to take, and why the bottom you find depends on where you start, with one chapter where you take the controls.

The surface is f(x, z) = 0.06(x² + z²) + 0.6 sin(1.2x) sin(z + 0.6) + 0.3 cos(0.8x − 0.7z) − 0.35 e^(−|p − c|²/2) with c = (−1.4, 1.1). Gradients are central differences; every run is computed in the page.

Helios · more films →