Entropy

Name Entropy

Description
Given

a finite set E of classified examples and

a set C = {c₁, ..., c_n} of possible classes.

Let |E_ci| denote the number of examples e ∈ E of class c_i and let p_i := |E_ci| / |E|.
Then the Entropy of E is defined as:
entropy(E) := - ∑ p_i * log₂(p_i).
_i=1,...,n

In this context we define 0 * log₂(0) = 0. The Entropy measures the impurity of a set of examples. It is lowest, if there is at most one class present, and it is highest, if the proportions of all present classes are equal.