Shannon Entropy Calculator (Information in Bits, Perplexity, Gini Impurity)
The entropy in bits of up to six outcomes from their frequencies or probabilities, with the maximum, the efficiency, the perplexity and the Gini impurity.
Results
What this tool does
Shannon entropy measures how unpredictable an outcome is, in bits: the average number of well-chosen yes/no questions needed to find out what happened. A fair coin has 1 bit, a fair die 2.585 bits, and a case with probabilities ½, ¼, ⅛, ⅛ has exactly 1.75 bits. Entropy is highest when all outcomes are equally likely and drops to zero when one outcome is certain. Perplexity, 2 raised to the entropy, is the number of equally likely outcomes that would be as unpredictable; Gini impurity, 1 − Σp², is the cousin used to split nodes in decision trees. Enter counts or percentages — they are normalised for you.
Formula
H = −Σ pᵢ log₂ pᵢ · H_max = log₂ k · perplexity = 2ᴴ · Gini impurity = 1 − Σ pᵢ²
Variables
| Symbol | Meaning | Unit |
|---|---|---|
w1 | Outcome 1 — frequency or probability | — |
w2 | Outcome 2 — frequency or probability | — |
w3 | Outcome 3 — frequency or probability | — |
w4 | Outcome 4 — frequency or probability | — |
w5 | Outcome 5 — frequency or probability | — |
w6 | Outcome 6 — frequency or probability | — |
H | Entropy (bits) | bits |
HM | Greatest possible entropy for that many outcomes | bits |
EF | Efficiency (entropy ÷ maximum) | % |
PP | Perplexity (equivalent equally likely outcomes) | — |
GI | Gini impurity | — |
K | Outcomes that can happen | — |
Worked example
- Outcome 1 — frequency or probability50
- Outcome 2 — frequency or probability25
- Outcome 3 — frequency or probability12.5
- Outcome 4 — frequency or probability12.5
- Outcome 5 — frequency or probability0
- Outcome 6 — frequency or probability0
- Entropy (bits)1.7500 bits
- Greatest possible entropy for that many outcomes2.0000 bits
- Efficiency (entropy ÷ maximum)87.5 %
- Perplexity (equivalent equally likely outcomes)3.364
- Gini impurity0.6563
- Outcomes that can happen4
Limitations
- The calculation runs at full precision and only the display is rounded. If you copy an intermediate value and retype it, small differences can appear.
- The default values are typical reference figures, not measurements of your situation. Replace them with your own data whenever you have it.