About Learning Rate Too High: Gradient Descent Divergence
This page shows what happens when the learning rate is too high. Gradient descent runs on the parabola f(x) = x squared with a learning rate of 1.05, just above the stability limit of 1. It is free, needs no sign-up and runs in your browser, so nothing you enter is uploaded.
On the parabola the gradient is 2x, so a step of plain gradient descent turns x into x(1 - 2 lr). With lr = 1.05 that factor is -1.1: x changes sign and grows by 10% every step, bouncing from side to side with ever larger swings. Below lr = 1 the factor is smaller than 1 in size, so x shrinks. At exactly lr = 1 it bounces between two values forever.
The other optimizers behave differently. RMSprop and Adam take steps of about the learning rate in size regardless of the gradient, so they cannot explode on this function, and momentum tolerates rates up to nearly 1.9 here. Lower the learning rate to 0.5 to see SGD land on the minimum in one step.
How to use Learning Rate Too High: Gradient Descent Divergence
- Choose a function. The suggested learning rate for it is filled in.
- Change the learning rate if you like. Smaller is slower and safer, larger is faster until it blows up.
- Press Step to move every optimizer once, or Play to animate. "+25 steps" skips ahead.
- Tap the plot to move the starting point (the cross). The paths restart from there.
- Read the table for each optimizer's position, value and status: running, converged or diverged.
- Watch the log chart: a line that keeps falling is converging, and a line that climbs is diverging.
Features
- Seven functions: three curves and four 2D surfaces, including a narrow valley, Rosenbrock, Himmelblau and a saddle.
- SGD, Momentum, RMSprop and Adam, all from the same start with the same learning rate.
- Contour map for 2D functions and a curve for 1D, with a coloured path for each optimizer.
- Divergence detection that stops a run and says which optimizers blew up.
- A log-scale chart of the function value against step.
- Step, play and +25 steps controls, with slower playback if you prefer reduced motion.
- Tap the plot to choose the starting point.
The divergence condition
For f(x) = x squared, one SGD step multiplies x by this factor. Divergence needs its absolute value to be above 1.
x_next = x (1 - 2 lr)
converges when |1 - 2 lr| < 1, that is 0 < lr < 1Worked examples
x^2, start 4, learning rate 1.05, 60 steps gives SGD diverged, Momentum 0.0255, RMSprop 0, Adam 0.0168
Plain SGD blows up. Momentum is still stable at this rate, because its velocity damps the overshoot and its limit is almost twice as high. RMSprop and Adam take steps of about the learning rate in size, so they settle near the minimum.
Privacy: how it works
- Everything is computed in your browser tab. Nothing is sent to Ulkaa or anyone else.
Tips
- On the parabola, try a learning rate of 0.5: SGD jumps to the minimum in a single step because the step size matches the curvature exactly.
- On the double well, drag the start to the left side and see it settle in the other, shallower minimum.
- On Himmelblau, change the start point and see different optimizers choose different minima.
- If a path vanishes off the plot, check the status column: it probably diverged.
Frequently asked questions
Is it really free?
Yes. There is no price, no trial and no credit card. Ulkaa Tool does the work with your own device, so there is no server bill to pass on to you.
Do I need an account?
No. There is no sign-up and no login. Open the page and use it. An Ulkaa account is not needed for anything on this tool.
Does it run in my browser?
Yes. Your settings are processed by code running in your browser tab, on your own device.
Is my data uploaded?
No. Your settings are never sent to Ulkaa or to anyone else. You can check this yourself by watching the Network tab in your browser developer tools while you use the tool.
Why does gradient descent diverge with a large learning rate?
The step overshoots the minimum by more than the distance it started from, so each step lands further away and the error grows.
What learning rate is too high?
It depends on the curvature of the function. For f(x) = x squared the limit is 1; for a steeper function the limit is lower.
More Gradient Descent Visualizer pages
- Gradient Descent VisualizerCompare SGD, Momentum, RMSprop and Adam side by side, step by step, on curves and contour maps.
- SGD Learning Rate VisualizerChange the learning rate and watch plain gradient descent speed up, oscillate or stall.
- Momentum vs SGDSee how a running velocity helps gradient descent in a long, narrow valley.
- Adam Optimizer VisualizerSee Adam adapt its step per direction on the curved Rosenbrock valley.
- RMSprop Optimizer VisualizerSee RMSprop normalise its steps on a surface with four minima.
- Gradient Descent and Local MinimaStart on one side of a double well and see which minimum each optimizer reaches.
- Gradient Descent at a Saddle PointStart almost on a ridge and see how long each optimizer takes to escape the saddle.
- Gradient Descent on the Rosenbrock FunctionFollow gradient descent down the curved banana valley of the Rosenbrock function.
- Gradient Descent on a Bumpy LossSee how a bumpy loss traps plain gradient descent and how momentum helps.