About the Cohen's Kappa Calculator
This Cohen's kappa calculator measures how well two raters agree when they classify the same items into categories, after removing the agreement you would expect by chance alone. Paste the agreement (confusion) table — rows for rater A's categories, columns for rater B's, in the same order — and it returns kappa (κ), the observed agreement, the chance-expected agreement, a standard error, a confidence interval, a z-test of κ = 0 and a Landis & Koch interpretation.
Kappa is the standard inter-rater reliability statistic in medicine (two clinicians reading the same scans), psychology and education (coding interview transcripts or marking essays), content analysis and machine-learning annotation. Raw percentage agreement overstates reliability when one category dominates; kappa corrects for that. For ordered categories such as severity grades, switch on linear or quadratic weights so that near-misses count as partial agreement.
Standard errors use the large-sample formulas of Fleiss, Cohen and Everitt (1969), which apply to both unweighted and weighted kappa: one version for the confidence interval and one computed under the null hypothesis for the z-test. Kappa assumes the same two raters rated every item independently; for three or more raters use Fleiss’ kappa.
With the default inputs, the cohen's kappa (κ) is 0.6828. Change any value above to recalculate instantly.
How to use the cohen's kappa calculator
- 1Cross-tabulate the two raters’ decisions: rows are rater A’s categories, columns are rater B’s, in the same order.
- 2Paste the counts, one row per line, separated by spaces or commas.
- 3Keep unweighted kappa for nominal categories, or choose linear or quadratic weights for ordered ones.
- 4Read kappa and its interpretation, then check the confidence interval for precision.
- 5Use the category table to see where the raters disagree most.
Formula and method
The table counts are divided by N to give proportions pᵢⱼ, with row totals pᵢ. (rater A) and column totals p.ⱼ (rater B). Observed agreement pₒ is the weighted share of items on which the raters agree, and chance agreement pₑ is what you would expect if both raters assigned categories independently at their own observed rates. Kappa rescales the improvement over chance so that 1 is perfect agreement and 0 is chance level; negative values mean less agreement than chance.
Unweighted kappa uses wᵢⱼ = 1 on the diagonal and 0 elsewhere. Linear weights are wᵢⱼ = 1 − |i − j| ÷ (k − 1) and quadratic weights are 1 − (i − j)² ÷ (k − 1)², which give partial credit for adjacent ordinal categories; quadratic-weighted kappa is closely related to the intraclass correlation. The standard error and z-test follow Fleiss, Cohen and Everitt (1969).
- pₒ
- Observed (weighted) proportion of agreement
- pₑ
- Agreement expected by chance from the marginal totals
- pᵢⱼ
- Proportion of items rated i by rater A and j by rater B
- wᵢⱼ
- Agreement weight for cell (i, j)
- k
- Number of categories
Worked examples
Two clinicians classify 100 cases into 3 diagnoses
The diagonal holds 25 + 30 + 24 = 79 agreements out of 100, so pₒ = 0.79. From the marginal totals (30, 40, 30 for rater A and 30, 38, 32 for rater B), chance agreement is 0.30×0.30 + 0.40×0.38 + 0.30×0.32 = 0.338. Kappa = (0.79 − 0.338) ÷ 0.662 = 0.683, substantial agreement, with a 95% CI of about 0.56 to 0.80.
Yes/no screening decision with a dominant category
The raters agree on 85% of items, which sounds high, but because both say "yes" most of the time the chance agreement is 0.85×0.90 + 0.15×0.10 = 0.78. Kappa is only (0.85 − 0.78) ÷ 0.22 = 0.32, fair agreement — a classic case where percent agreement is misleading.
Severity grades 1–4 with quadratic weights
Most disagreements are between adjacent grades, so quadratic weights treat them as near-misses (weight 0.889 for a one-grade gap, 0.556 for two). Weighted kappa is 0.77, substantial agreement, while the unweighted kappa for the same table is only 0.54 because it counts every near-miss as a full disagreement.
Frequently asked questions
What is a good Cohen's kappa value?+
Landis and Koch (1977) suggested: below 0 poor, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect. Many fields expect at least 0.60 for research coding, and higher for clinical decisions.
Why is kappa lower than percent agreement?+
Kappa subtracts the agreement two raters would reach by chance given how often each uses each category. When one category is very common, chance agreement is high, so even a high raw agreement can produce a modest kappa.
When should I use weighted kappa?+
Use weighted kappa when categories are ordered, such as mild/moderate/severe or a 1–5 rating. Linear weights give partial credit in proportion to the distance; quadratic weights penalize large disagreements more heavily and are the most commonly reported.
Can kappa be negative?+
Yes. A negative kappa means the raters agree less often than chance would predict, which usually signals a systematic problem such as reversed coding or misunderstood category definitions.
What is the difference between Cohen's and Fleiss' kappa?+
Cohen's kappa compares exactly two raters who each rate every item. Fleiss' kappa extends the idea to three or more raters, and it allows different raters to rate different items.
How do I lay out the table?+
Each row is a category chosen by rater A and each column the same category chosen by rater B. Cell (i, j) counts items rater A put in category i and rater B put in category j, so agreements are on the diagonal.