About Activation Function Plotter
This activation function plotter is free, needs no sign-up and runs in your browser. Choose sigmoid, tanh, ReLU, leaky ReLU, ELU, GELU, swish (SiLU), softplus or softmax. You get the formula and its derivative in LaTeX, an interactive plot of both curves you can zoom and trace, a value at any x you type, and a table of values.
Softmax gets its own mode because it works on a whole vector: type a list of scores, optionally a temperature, and see the probabilities with every step of the numerically stable calculation. A compare view overlays all the other functions so that you can see how their shapes differ.
Activation functions are what make a neural network more than a stack of linear maps. The shape of the curve and of its derivative decide how well gradients flow in training: a function that flattens out (sigmoid, tanh) shrinks the learning signal in deep networks, while ReLU and its relatives keep a gradient of one on the positive side.
How to use Activation Function Plotter
- Choose a function from the menu, or "Compare all" to overlay the curves.
- For leaky ReLU, ELU and swish, set the extra parameter (alpha or beta) if you want something other than the default.
- Type an x to see f(x) and f'(x), and read the table of values.
- Zoom, pan and trace the plot to read the curve. Double-tap resets the view.
- For softmax, enter your scores separated by commas, set the temperature and read the probabilities and the steps.
- Read the strengths and weaknesses under the plot to compare the functions.
Features
- Nine activation functions with formulas and derivatives rendered in LaTeX.
- Interactive plot of the function and its derivative, with zoom, pan, trace and PNG download.
- A compare view of eight functions on one plot.
- Parameters for leaky ReLU (alpha), ELU (alpha) and swish (beta).
- GELU computed exactly with the error function, not the tanh approximation.
- Softmax with temperature and a worked, numerically stable calculation.
- A value table, range, and plain-language strengths and weaknesses for each function.
Formulas
Each function is applied to a single number, except softmax, which acts on a vector of scores. The derivative is what backpropagation multiplies by on the way back through the layer, so its size controls how fast a network can learn.
sigmoid(x) = 1 / (1 + e^-x)
tanh(x) = (e^x - e^-x) / (e^x + e^-x)
ReLU(x) = max(0, x)
LeakyReLU(x) = x if x > 0 else alpha x
ELU(x) = x if x > 0 else alpha (e^x - 1)
GELU(x) = x Phi(x)
swish(x) = x sigmoid(beta x)
softplus(x) = ln(1 + e^x)
softmax(z)_i = e^z_i / sum of e^z_jWorked examples
sigmoid at x = 0, and its derivative gives 0.5 and 0.25
The sigmoid is exactly 0.5 at the origin, and its derivative peaks there at 0.25, so gradients through a sigmoid are always cut to at most a quarter.
GELU at x = 1, and swish at x = 1 gives 0.8413 and 0.7311
Both sit below the input itself and approach it as x grows, which is what makes them smooth versions of ReLU.
softmax of the scores 2, 1, 0.1 gives 0.659, 0.2424, 0.0986
The three probabilities add up to 1. The score 2 gets 66% because e^2 is much larger than e^1 and e^0.1.
Choosing an activation function
| Function | Range | Common use | Watch out for |
|---|---|---|---|
| Sigmoid | (0, 1) | Binary output, gates | Vanishing gradients |
| Tanh | (-1, 1) | Recurrent networks | Saturation |
| ReLU | [0, infinity) | Default hidden layers | Dead units |
| Leaky ReLU | (-infinity, infinity) | When ReLU units die | Extra setting |
| ELU | (-alpha, infinity) | Smooth hidden layers | Cost of the exponential |
| GELU | [about -0.17, infinity) | Transformers | Cost |
| Swish (SiLU) | [about -0.28, infinity) | Deep convolutional and transformer models | Cost |
| Softplus | (0, infinity) | Positive outputs, smooth ReLU | Never exactly zero |
| Softmax | (0, 1), sums to 1 | Multi-class output layer | Overconfidence |
Good to know
Vanishing gradients and dead units
The derivative of the sigmoid never exceeds 0.25 and falls toward zero for large positive or negative inputs. Multiply several such factors through many layers and the learning signal for early layers becomes tiny. Tanh has a peak slope of 1 but also saturates. ReLU has a slope of exactly 1 for positive inputs, which is a main reason it made deep networks easy to train.
ReLU's weakness is the other side: with a negative input its slope is 0, so a unit whose input is always negative stops learning, a dead ReLU. Leaky ReLU, ELU and GELU keep a small non-zero slope there. At exactly zero ReLU has no derivative, and software uses a convention; this tool uses 0.
Softmax, temperature and stability
Softmax exponentiates every score and divides by the total. A direct calculation overflows for large scores, so the standard method first subtracts the largest score, which leaves the result unchanged. Dividing the scores by a temperature before the softmax changes how peaked the output is: a high temperature pushes probabilities toward equal, a low one toward a single 1.
A softmax output looks like a probability but is not guaranteed to be calibrated. A trained network is often overconfident, so a 99% output does not mean the prediction is right 99 times out of 100.
Privacy: how it works
- All calculations and plots run in your browser tab. Nothing is sent to Ulkaa or anyone else.
Tips
- Compare the derivative curves: where one is flat near zero, learning is slow in that region of input.
- Try a leaky ReLU with a large alpha such as 0.3 to see how the negative side tilts, and a swish beta of 5 to see it approach ReLU.
- For softmax, add 100 to every score: the probabilities do not change, because only the differences matter.
- Scores with one very large value give a probability close to 1 for that class, whatever the others are.
Frequently asked questions
Is it really free?
Yes. There is no price, no trial and no credit card. Ulkaa Tool does the work with your own device, so there is no server bill to pass on to you.
Do I need an account?
No. There is no sign-up and no login. Open the page and use it. An Ulkaa account is not needed for anything on this tool.
Does it run in my browser?
Yes. Your inputs are processed by code running in your browser tab, on your own device.
Is my data uploaded?
No. Your inputs are never sent to Ulkaa or to anyone else. You can check this yourself by watching the Network tab in your browser developer tools while you use the tool.
What is an activation function?
A function applied to the output of a neuron so that a network can represent more than straight-line relationships. Without one, any stack of layers would collapse to a single linear map.
What is the derivative of the sigmoid?
sigmoid(x) times one minus sigmoid(x). It peaks at 0.25 when x is 0 and falls toward zero on both sides.
Why is ReLU so popular?
It is very cheap, has a gradient of 1 for positive inputs so it avoids vanishing gradients, and works well in deep networks.
What is the difference between GELU and ReLU?
GELU is smooth and lets small negative inputs through, weighting the input by the normal cumulative distribution. ReLU has a sharp corner at zero and outputs zero for every negative input.
Is swish the same as SiLU?
SiLU is swish with beta equal to 1: x times sigmoid(x). Swish allows other values of beta.
How does softmax turn scores into probabilities?
It exponentiates each score and divides by the sum of all the exponentials, so the results are positive and add up to 1.
What does temperature do in softmax?
The scores are divided by the temperature first. A temperature above 1 flattens the distribution and below 1 sharpens it.
Does the tool use the tanh approximation of GELU?
No. It uses the exact definition with the error function. The common tanh approximation differs by less than about 0.001.
Popular Activation Function Plotter pages
- Sigmoid Function CalculatorEvaluate the logistic sigmoid and its derivative at any x, and plot both.
- Tanh Function CalculatorEvaluate tanh and its derivative at any x, and plot both.
- ReLU Activation FunctionPlot max(0, x) and its derivative and evaluate them at any input.
- Leaky ReLU CalculatorSet the negative slope alpha and see how leaky ReLU differs from ReLU.
- ELU Activation FunctionPlot the exponential linear unit and its derivative for any alpha.
- GELU Activation FunctionPlot the Gaussian error linear unit and its derivative, and evaluate it at any x.
- Swish (SiLU) Activation FunctionPlot x times sigmoid(beta x) and its derivative, and change beta.
- Softplus Function CalculatorPlot softplus and its derivative, which is exactly the sigmoid.
- Softmax CalculatorTurn a list of scores into probabilities, with the working shown.