 Cross entropy

In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p.
The cross entropy for two distributions p and q over the same probability space is thus defined as follows:
 ,
where H(p) is the entropy of p, and D_{KL}(p   q) is the KullbackLeibler divergence of q from p (also known as the relative entropy).
For discrete p and q this means
The situation for continuous distributions is analogous:
NB: The notation H(p,q) is sometimes used for both the cross entropy as well as the joint entropy of p and q.
Estimation
There are many situations where crossentropy needs to be measured but the distribution of p is unknown. An example is language modeling, where a model is created based on a training set T, and then its crossentropy is measured on a test set to assess how accurate the model is in predicting the test data. In this example, p is the true distribution of words in any corpus, and q is the distribution of words as predicted by the model. Since the true distribution is unknown, crossentropy cannot be directly calculated. In these cases, an estimate of crossentropy is calculated using the following formula:
where N is the size of the test set, and q(x) is the probability of event x estimated from the training set. It should be noted that the sum is calculated over N.
Crossentropy minimization
Crossentropy minimization is frequently used in optimization and rareevent probability estimation; see the crossentropy method.
When comparing a distribution q against a fixed reference distribution p, cross entropy and KL divergence are identical up to an additive constant (since p is fixed): both take on their minimal values when p = q, which is 0 for KL divergence, and H(p) for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum CrossEntropy (MCE), or Minxent.
However, as discussed in the article KullbackLeibler divergence, sometimes the distribution q is the fixed prior reference distribution, and the distribution p is optimised to be as close to q as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining crossentropy to be D_{KL}(pq) , rather than H(p,q).
See also
Categories: Entropy and information
Wikimedia Foundation. 2010.
Look at other dictionaries:
Crossentropy method — The cross entropy (CE) method attributed to Reuven Rubinstein is a general Monte Carlo approach to combinatorial and continuous multi extremal optimization and importance sampling. The method originated from the field of rare event simulation,… … Wikipedia
Entropy (information theory) — In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information… … Wikipedia
The Way of Cross and Dragon — is a science fiction short story by George R. R. Martin. It involves a far future priest of the One True Interstellar Catholic Church of Earth and the Thousand Worlds (with similarities to the Roman Catholic hierarchy) investigating a sect that… … Wikipedia
Kullback–Leibler divergence — In probability theory and information theory, the Kullback–Leibler divergence[1][2][3] (also information divergence, information gain, relative entropy, or KLIC) is a non symmetric measure of the difference between two probability distributions P … Wikipedia
Méthode de l'entropie croisée — Traduction à relire Cross entropy method → … Wikipédia en Français
List of mathematics articles (C) — NOTOC C C closed subgroup C minimal theory C normal subgroup C number C semiring C space C symmetry C* algebra C0 semigroup CA group Cabal (set theory) Cabibbo Kobayashi Maskawa matrix Cabinet projection Cable knot Cabri Geometry Cabtaxi number… … Wikipedia
Perplexity — is a measurement in information theory. It is defined as 2 raised to the power of entropy, or more often as 2 raised to the power of cross entropy. The latter definition is commonly used to compare probability models empirically. Perplexity of a… … Wikipedia
Information theory — Not to be confused with Information science. Information theory is a branch of applied mathematics and electrical engineering involving the quantification of information. Information theory was developed by Claude E. Shannon to find fundamental… … Wikipedia
Quantities of information — A simple information diagram illustrating the relationships among some of Shannon s basic quantities of information. The mathematical theory of information is based on probability theory and statistics, and measures information with several… … Wikipedia
Prior probability — Bayesian statistics Theory Bayesian probability Probability interpretations Bayes theorem Bayes rule · Bayes factor Bayesian inference Bayesian network Prior · Posterior · Likelihood … Wikipedia