About Cohen's Kappa Calculator
This Cohen's kappa calculator takes a square agreement matrix, such as two raters labelling the same items, and reports kappa with the observed and the chance agreement shown. It is free, needs no sign-up and runs in your browser, so nothing you enter is uploaded.
Two raters who say yes 90% of the time will agree often just by chance. Kappa removes that chance agreement: it is the observed agreement minus the expected agreement, divided by how much room above chance there was.
Landis and Koch suggested reading values above 0.8 as almost perfect, 0.6 to 0.8 as substantial, 0.4 to 0.6 as moderate and below 0.2 as slight. These bands are conventions, not laws. In the preloaded example two raters agree on 35 of 50 items, but chance alone would give 25.
How to use Cohen's Kappa Calculator
- Choose "Matrix counts" and set the number of classes, or choose "Actual and predicted labels" and paste the two lists.
- In matrix mode, rows are the actual class and columns the predicted class. Edit the cells and the class names.
- Pick the headline metric: accuracy, precision, recall, specificity, F1, F-beta, MCC, kappa or balanced accuracy.
- For precision, recall and F-scores, choose which class counts as positive. With two classes the first class is the positive one.
- Read the headline value and its explanation, then scroll to the per-class table, the averages and the worked steps.
- Use the copy button to take the metrics into your notes or report.
Features
- Binary and multi-class matrices up to 8 classes, or up to 12 classes built from pasted labels.
- Accuracy, precision, recall, specificity, F1, F-beta, MCC, Cohen's kappa and balanced accuracy in one place.
- Per-class true positive, false positive, false negative and true negative counts.
- Macro, micro and weighted averages for multi-class problems, so you can see how each differs.
- Worked steps with your numbers substituted into every formula, and the formulas rendered in LaTeX.
- A ratio with a zero denominator is shown as n/a instead of a misleading 0.
- Plain-English reading of the headline metric, including a warning about accuracy on imbalanced data.
- Editable class names, so the output reads like your own problem.
Kappa formula
p_o is the observed agreement (the diagonal share) and p_e is the agreement expected if the raters were independent.
kappa = (po - pe) / (1 - pe)
pe = sum over classes of (row total x column total) / N^2Worked examples
Rater A yes/no versus rater B: 20, 5, 10, 15 gives kappa 0.4
Observed agreement is 0.7. Expected agreement is 0.5, so kappa is (0.7 - 0.5) / 0.5 = 0.4.
Privacy: how it works
- The matrix and labels you enter are processed by code running in your browser tab. They are not sent to Ulkaa or anyone else.
Tips
- Do not rely on accuracy alone when one class is much rarer than the others. Check recall for that class and the MCC.
- Precision and recall trade against each other. Moving a classifier's decision threshold raises one and lowers the other; the ROC and AUC calculator shows the whole trade-off.
- F1 gives equal weight to precision and recall. Use F-beta with beta above 1 when missing a positive is worse than a false alarm, and below 1 when the reverse is true.
- A metric with an empty denominator, such as precision for a class never predicted, is shown as n/a and left out of macro averages.
Frequently asked questions
Is it really free?
Yes. There is no price, no trial and no credit card. Ulkaa Tool does the work with your own device, so there is no server bill to pass on to you.
Do I need an account?
No. There is no sign-up and no login. Open the page and use it. An Ulkaa account is not needed for anything on this tool.
Does it run in my browser?
Yes. Your matrix and labels are processed by code running in your browser tab, on your own device.
Is my data uploaded?
No. Your matrix and labels are never sent to Ulkaa or to anyone else. You can check this yourself by watching the Network tab in your browser developer tools while you use the tool.
What does Cohen's kappa measure?
Agreement between two raters, or a model and the truth, corrected for the agreement expected by chance. 0 is chance level and 1 is perfect agreement.
Why is kappa lower than plain agreement?
Because some agreement happens by luck, especially when one answer is very common. Kappa subtracts that luck.
More Confusion Matrix Calculator pages
- Confusion Matrix CalculatorEnter a confusion matrix, or paste actual and predicted labels, and get every classification metric with the working.
- Precision and Recall CalculatorEnter your confusion matrix to get precision and recall with the formula worked through.
- F1 Score CalculatorGet the F1 score of a classifier from its confusion matrix, with every step shown.
- Classification Accuracy CalculatorWork out accuracy from a confusion matrix and see when it misleads.
- MCC Calculator (Matthews Correlation Coefficient)Compute the MCC of a classifier from its confusion matrix, with the formula substituted.
- Sensitivity and Specificity CalculatorEnter a test's counts to get sensitivity (recall) and specificity, with the working shown.
- F-beta Score CalculatorChoose beta to weight recall or precision more, and get the F-beta score with its working.
- Balanced Accuracy CalculatorAverage the recall of every class so a rare class counts as much as a common one.
- Multi-class Confusion Matrix CalculatorScore a classifier with three or more classes, class by class and on average.
- Confusion Matrix From Actual and Predicted LabelsPaste the true labels and the predictions and get the matrix and its metrics.