University / ProfessionalLessonCalculus

Gradient Descent: How Machines Learn

The one idea behind most of machine learning: roll downhill. Interactive 3D — orbit the landscape, drop the ball anywhere.

dailymath · July 18, 2026 · 9 min read

The whole trick: roll downhill

Every neural network — the ones behind chatbots, image generators, recommendation feeds — learns by the same move repeated billions of times: measure how wrong you are, then nudge every knob a tiny bit in the direction that makes you less wrong. That's it. The rest is bookkeeping.

Mathematicians call the wrongness a cost function. Training means finding the knob settings where the cost is smallest — and the tool for that is calculus you already know: the derivative.

One dimension first

Take

g(x) = x^2

. Wherever you stand, the derivative

g'(x) = 2x

tells you the slope. Walk *against* the slope and you must go down. Gradient descent formalizes the walk:

The update rule

x_{k+1} = x_k - \eta \, g'(x_k)

The number

\eta

(eta) is the learning rate — the step size. Watch the ball take the walk on

g(x) = x^2

: each step lands where the previous slope pointed, and the steps shrink as the slope flattens.

Figure 1

Ten steps of

x_{k+1} = x_k - \eta\,g'(x_k)

from

x_0 = -1.8

with

\eta = 0.15

. Steps shrink automatically as the slope flattens — the derivative is both compass and speedometer.

Now in 3D — the real thing

Real cost functions have millions of dimensions, but everything important already shows up in two. The surface below is

f(x, y) = (x^2-1)^2 + 1.2y^2

: two valleys separated by a mountain pass (a saddle point). The gradient

\nabla f

is now a vector of partial derivatives, and the update rule is the same walk:

Two dimensions

\begin{pmatrix} x_{k+1} \\ y_{k+1} \end{pmatrix} = \begin{pmatrix} x_k \\ y_k \end{pmatrix} - \eta \begin{pmatrix} \partial f/\partial x \\ \partial f/\partial y \end{pmatrix}

Figure 2 · Interactive

Drag to orbit the landscape (two fingers on a phone). Tap anywhere to drop the ball and watch it follow

-\nabla f

downhill. Notice: where you drop it decides which valley it ends up in.

Local minima

The tap experiment is the deepest lesson in machine learning: gradient descent finds *a* valley, not *the* valley. Start on the wrong side of the pass and you'll settle into a different answer. Modern ML mostly makes peace with this — good-enough valleys are everywhere in high dimensions.

The Goldilocks knob

Everything hangs on

\eta

. Too small and training takes forever; too large and each step overshoots the valley floor, bouncing higher and higher until the whole thing explodes. Practitioners spend real money finding the porridge that's just right.

Figure 3

Same curve, same start, three learning rates. The divergent case isn't exotic — it's what a training run looks like when the loss goes to `NaN`.

Why saddle points matter

At the mountain pass

(0,0)

the gradient is exactly zero — the walk stops even though you're nowhere near a minimum. In millions of dimensions, saddle points vastly outnumber minima, and escaping them (with momentum, noise, or clever step sizes) is a whole subfield. When you hear "the optimizer got stuck", picture the ball balanced on that pass.

From here the story branches into stochastic gradient descent (estimate the gradient from a random handful of data), momentum (give the ball inertia), and Adam (give every knob its own learning rate). All of them are still, at heart, the walk you just watched.

Check your intuition

Question 1 of 4

In the update rule

x_{k+1} = x_k - \eta\,g'(x_k)

, why is the derivative *subtracted*?

Gradient descent is the whole trick behind machine learning, run backward from the answer: measure the wrongness, follow the slope down. If chasing that kind of intuition down to the arithmetic is your thing, solve a fresh math problem every morning with dailymath. One drop, one coffee, one win.

Try dailymath

Back to posts