About Gradient Descent Visualizer
This gradient descent visualizer is free, needs no sign-up and runs in your browser. Pick a function, set a learning rate, and watch four optimizers (SGD, Momentum, RMSprop and Adam) start from the same point and follow the slope downhill. Step one move at a time or press Play, and tap the plot to move the starting point.
One-variable functions are drawn as curves and two-variable functions as contour maps, with the path of each optimizer in its own colour. A table lists each optimizer's position and value, and a log-scale chart shows how fast each one lowers the function.
Gradient descent is how nearly every neural network is trained, and the learning rate is the setting that most often goes wrong. Raise it far enough and you can watch an optimizer diverge, with the error growing instead of shrinking. Try the narrow valley, the Rosenbrock banana or the saddle to see why adaptive methods exist.
How to use Gradient Descent Visualizer
- Choose a function. The suggested learning rate for it is filled in.
- Change the learning rate if you like. Smaller is slower and safer, larger is faster until it blows up.
- Press Step to move every optimizer once, or Play to animate. "+25 steps" skips ahead.
- Tap the plot to move the starting point (the cross). The paths restart from there.
- Read the table for each optimizer's position, value and status: running, converged or diverged.
- Watch the log chart: a line that keeps falling is converging, and a line that climbs is diverging.
Features
- Seven functions: three curves and four 2D surfaces, including a narrow valley, Rosenbrock, Himmelblau and a saddle.
- SGD, Momentum, RMSprop and Adam, all from the same start with the same learning rate.
- Contour map for 2D functions and a curve for 1D, with a coloured path for each optimizer.
- Divergence detection that stops a run and says which optimizers blew up.
- A log-scale chart of the function value against step.
- Step, play and +25 steps controls, with slower playback if you prefer reduced motion.
- Tap the plot to choose the starting point.
The update rules
Plain gradient descent moves against the gradient by a step of size equal to the learning rate times the slope. The other three change what is multiplied.
Momentum keeps a running velocity so steps build up along a consistent direction and damp out in directions that flip sign. RMSprop divides each step by a running average of recent squared gradients, so steep directions get smaller steps. Adam combines both and corrects the bias of its running averages at the start.
SGD: x = x - lr g
Momentum: v = 0.9 v + g; x = x - lr v
RMSprop: s = 0.9 s + 0.1 g^2; x = x - lr g / (sqrt(s) + eps)
Adam: m = 0.9 m + 0.1 g; s = 0.999 s + 0.001 g^2
x = x - lr (m / (1 - 0.9^t)) / (sqrt(s / (1 - 0.999^t)) + eps)Worked examples
Narrow valley x^2 + 10 y^2, learning rate 0.08, 40 steps gives SGD f = 0, Adam f = 0.3647
With a well-chosen rate, plain SGD does very well on this smooth bowl: its step across the steep y direction flips sign each time (a factor of -0.6) but shrinks fast. Adam and RMSprop move about one learning rate per step, so they hover near the minimum instead of landing on it. Raise the rate to 0.15 and SGD diverges while RMSprop and Adam still settle.
Parabola x^2 with SGD and learning rate 1.05 gives SGD diverged after 58 steps
Each step multiplies x by 1 - 2 x 1.05 = -1.1, so x flips sign and grows by 10% every time, and f grows by 21%. Below a learning rate of 1 the same rule shrinks x.
Which optimizer for which landscape
| Optimizer | Strength | Weakness |
|---|---|---|
| SGD | Simple, few settings | Slow in narrow valleys, sensitive to learning rate |
| Momentum | Builds speed along a steady slope | Can overshoot and oscillate |
| RMSprop | Adapts step size per direction | No momentum, step size still needs tuning |
| Adam | Fast start, robust defaults | Can generalise slightly worse than tuned SGD |
Good to know
Why a high learning rate diverges
Near a minimum a function looks like a bowl with some curvature. A gradient step overshoots the bottom when the learning rate times the curvature is above 2, and each overshoot lands further up the opposite wall. The error then grows geometrically. A steeper function therefore needs a smaller learning rate, which is why a narrow valley is stable only for rates below the steep direction's limit.
What the demo does not show
These are exact gradients of small test functions. Real training uses noisy gradients from batches of data (the S in SGD), many millions of parameters, learning-rate schedules and often weight decay. The shapes here explain the behaviour; they are not a benchmark of the optimizers on real models.
Privacy: how it works
- Everything is computed in your browser tab. Nothing is sent to Ulkaa or anyone else.
Tips
- On the parabola, try a learning rate of 0.5: SGD jumps to the minimum in a single step because the step size matches the curvature exactly.
- On the double well, drag the start to the left side and see it settle in the other, shallower minimum.
- On Himmelblau, change the start point and see different optimizers choose different minima.
- If a path vanishes off the plot, check the status column: it probably diverged.
Frequently asked questions
Is it really free?
Yes. There is no price, no trial and no credit card. Ulkaa Tool does the work with your own device, so there is no server bill to pass on to you.
Do I need an account?
No. There is no sign-up and no login. Open the page and use it. An Ulkaa account is not needed for anything on this tool.
Does it run in my browser?
Yes. Your settings are processed by code running in your browser tab, on your own device.
Is my data uploaded?
No. Your settings are never sent to Ulkaa or to anyone else. You can check this yourself by watching the Network tab in your browser developer tools while you use the tool.
What is gradient descent?
An algorithm that finds a minimum of a function by repeatedly stepping in the direction of steepest descent, which is against the gradient, by an amount set by the learning rate.
What is the learning rate?
The size of each step relative to the gradient. Too small and progress is slow; too large and the steps overshoot, and the optimizer can diverge.
Why does my optimizer diverge?
The learning rate is too high for the curvature of the function, so each step overshoots by more than it started from and the error grows.
What does momentum do?
It accumulates a velocity from past gradients, so steps build up along a consistent direction and cancel in directions that flip. It speeds travel along valleys and can roll through shallow bumps.
What is the difference between RMSprop and Adam?
Both scale the step by a running average of squared gradients. Adam also keeps a running average of the gradient itself (momentum) and corrects the bias of both averages early in training.
Is Adam always the best optimizer?
No. It is a strong default, but a well-tuned SGD with momentum often generalises as well or better on some tasks. The right choice depends on the problem.
Does this use real neural-network training?
No. It optimises fixed mathematical functions so you can see the paths clearly. Real training adds noisy mini-batch gradients and millions of parameters.
Popular Gradient Descent Visualizer pages
- Learning Rate Too High: Gradient Descent DivergenceSee the error grow instead of shrink when the learning rate crosses its limit.
- SGD Learning Rate VisualizerChange the learning rate and watch plain gradient descent speed up, oscillate or stall.
- Momentum vs SGDSee how a running velocity helps gradient descent in a long, narrow valley.
- Adam Optimizer VisualizerSee Adam adapt its step per direction on the curved Rosenbrock valley.
- RMSprop Optimizer VisualizerSee RMSprop normalise its steps on a surface with four minima.
- Gradient Descent and Local MinimaStart on one side of a double well and see which minimum each optimizer reaches.
- Gradient Descent at a Saddle PointStart almost on a ridge and see how long each optimizer takes to escape the saddle.
- Gradient Descent on the Rosenbrock FunctionFollow gradient descent down the curved banana valley of the Rosenbrock function.
- Gradient Descent on a Bumpy LossSee how a bumpy loss traps plain gradient descent and how momentum helps.