Differential entropy

Differential entropy

Differential entropy (also referred to as continuous entropy) is a concept in information theory that extends the idea of (Shannon) entropy, a measure of average surprisal of a random variable, to continuous probability distributions.



Let X be a random variable with a probability density function f whose support is a set \mathbb X. The differential entropy h(X) or h(f) is defined as

h(X) = -\int_\mathbb{X} f(x)\log f(x)\,dx.

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, Uniform(0,1/2) has negative differential entropy \int_0^\frac{1}{2} -2\log2\,dx=-\log2\,.

Thus, differential entropy does not share all properties of discrete entropy.

Note that the continuous mutual information I(X;Y) has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of X and Y as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps) [1], including linear [2] transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

Properties of differential entropy

  • For two densities f and g, D(f||g) \geq 0 with equality if f = g almost everywhere. Similarly, for two random variables X and Y, I(X;Y) \geq 0 and h(X|Y) \leq h(X) with equality if and only if X and Y are independent.
  • The chain rule for differential entropy holds as in the discrete case
h(X_1, \ldots, X_n) = \sum_{i=1}^{n} h(X_i|X_1, \ldots, X_{i-1}) \leq \sum h(X_i).
  • Differential entropy is translation invariant, ie, h(X + c) = h(X) for a constant c.
  • Differential entropy is in general not invariant under arbitrary invertible maps. In particular, for a constant a, h(aX) = h(X) + \log \left| a \right|. For a vector valued random variable X and a matrix A, h(A\mathbf{X}) = h(\mathbf{X}) + \log \left| \det A \right|.
  • In general, for a transformation from a random vector X to a random vector with same dimension Y \mathbf{Y} = m(\mathbf{X}), the corresponding entropies are related via h(\mathbf{Y}) \leq h(\mathbf{X}) + \int f(x) \log \left\vert \frac{\partial m}{\partial x} \right\vert dx where \left\vert \frac{\partial m}{\partial x} \right\vert is the Jacobian of the transformation m. Equality is achieved if the transform is bijective, i.e., invertible.
  • If a random vector \mathbf{X} \in \mathbb{R}^{n} has mean zero and covariance matrix K, h(\mathbf{X}) \leq \frac{1}{2} \log[(2\pi e)^n \det{K}] with equality if and only if X is jointly gaussian.

However, differential entropy does not have other desirable properties:

A modification of differential entropy that addresses this is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distribution

With a normal distribution, differential entropy is maximized for a given variance. The following is a proof that a Gaussian variable has the largest entropy amongst all random variables of equal variance.

Let g(x) be a Gaussian PDF with mean μ and variance σ2 and f(x) an arbitrary PDF with the same variance. Since differential entropy is translation invariant we can assume that f(x) has the same mean of μ as g(x).

Consider the Kullback-Leibler divergence between the two distributions

 0 \leq D_{KL}(f || g) &= \int_{-\infty}^\infty f(x) \log \left( \frac{f(x)}{g(x)} \right) dx \\
 &= -h_{f(x)} - \int_{-\infty}^\infty f(x)\log(g(x)) dx.

Now note that

 \int_{-\infty}^\infty f(x)\log(g(x)) dx &= \int_{-\infty}^\infty f(x)\log\left( \frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\right) dx \\
 &= \int_{-\infty}^\infty f(x) \log\frac{1}{\sqrt{2\pi\sigma^2}} dx + \int_{-\infty}^\infty f(x)\left( -\frac{(x-\mu)^2}{2\sigma^2}\right) dx \\
 &= -\frac{1}{2}\log(2\pi\sigma^2) - \frac{\sigma^2}{2\sigma^2} = -\frac{1}{2}\left(\log(2\pi\sigma^2) + 1\right) \\
 &= -h_{g(x)} 

because the result does not depend on f(x) other than through the variance. Combining the two results yields

 h_{g(x)} - h_{f(x)} \geq 0 \!

with equality when g(x) = f(x) following from the properties of Kullback-Leibler divergence.

This result may also be demonstrated using the variational calculus. A Lagrangian function with two Lagrangian multipliers may be defined as:

L=\int_{-\infty}^\infty g(x)\ln(g(x))\,dx+\lambda_0\left(1-\int_{-\infty}^\infty g(x)\,dx\right)+\lambda\left(\sigma^2-\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right)

where g(x) is some function with mean μ. When the entropy of g(x) is at a maximum and the constraint equations, which consist of the normalization condition \left(1=\int_{-\infty}^\infty g(x)\,dx\right) and the requirement of fixed variance \left(\sigma^2=\int_{-\infty}^\infty g(x)(x-\mu)^2\,dx\right), are both satisfied, then a small variation δg(x) about g(x) will produce a variation δL about L which is equal to zero:

0=\delta L=\int_{-\infty}^\infty \delta g(x)\left[\ln(g(x))+1+\lambda_0+\lambda(x-\mu)^2\right]\,dx

Since this must hold for any small δg(x), the term in brackets must be zero, and solving for g(x) yields:


Using the constraint equations to solve for λ0 and λ yields the normal distribution:


Example: Exponential distribution

Let X be an exponentially distributed random variable with parameter λ, that is, with probability density function

f(x) = \lambda e^{-\lambda x} \mbox{ for } x \geq 0.

Its differential entropy is then

h_e(X)\, =-\int_0^\infty \lambda e^{-\lambda x} \log (\lambda e^{-\lambda x})\,dx
=  -\left(\int_0^\infty (\log \lambda)\lambda e^{-\lambda x}\,dx + \int_0^\infty (-\lambda x) \lambda e^{-\lambda x}\,dx\right)
= -\log \lambda \int_0^\infty f(x)\,dx + \lambda E[X]
= -\log\lambda + 1\,.

Here, he(X) was used rather than h(X) to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Differential entropies for various distributions

In the table below, \Gamma(x) = \int_0^{\infty} e^{-t} t^{x-1} dt (the gamma function), \psi(x) = \frac{d}{dx} \Gamma(x), B(p,q) = \frac{\Gamma(p)\Gamma(q)}{\Gamma(p+q)}, and γE is Euler's constant. Each distribution maximizes the entropy for a particular set of functional constraints listed in the fourth column, and the constraint that x be included in the support of the probability density, which is listed in the fifth column[3].

Table of differential entropies and corresponding maximum entropy constraints
Distribution Name Probability density function (pdf) Entropy in nats Maximum Entropy Constraint Support
Uniform f(x) = \frac{1}{b-a} \ln(b - a) \, None [a,b]\,
Normal f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) \ln\left(\sigma\sqrt{2\,\pi\,e}\right) E(x)=\mu,\,E((x-\mu)^2)=\sigma^2 (-\infty,\infty)\,
Exponential f(x) = \lambda \exp\left(-\lambda x\right) 1 - \ln \lambda \, E(x)=1/\lambda\, [0,\infty)\,
Rayleigh f(x) = \frac{x}{\sigma^2} \exp\left(-\frac{x^2}{2\sigma^2}\right) 1 + \ln \frac{\sigma}{\sqrt{2}} + \frac{\gamma_E}{2} E(x^2)=2\sigma^2, E(\ln(x))=\frac{\ln(2\sigma^2)-\gamma_E}{2}\, [0,\infty)\,
Beta f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha,\beta)} for 0 \leq x \leq 1  \ln B(\alpha,\beta) - (\alpha-1)[\psi(\alpha) - \psi(\alpha +\beta)]\,
- (\beta-1)[\psi(\beta) - \psi(\alpha + \beta)] \,
E(\ln(1-x))=\psi(\beta )-\psi(\alpha+\beta)\,
Cauchy f(x) = \frac{\gamma}{\pi} \frac{1}{\gamma^2 + x^2} \ln(4\pi\gamma) \, E(\ln(x^2+\gamma^2))=\ln(4\gamma^2)\, (-\infty,\infty)\,
Chi f(x) = \frac{2}{2^{k/2}  \Gamma(k/2)} x^{k-1} \exp\left(-\frac{x^2}{2}\right) \ln{\frac{\Gamma(k/2)}{\sqrt{2}}} - \frac{k-1}{2} \psi\left(\frac{k}{2}\right) + \frac{k}{2} E(x^2)=k,\,E(\ln(x))=\frac{1}{2}\left[\psi\left(\frac{k}{2}\right)\!+\!\ln(2)\right] [0,\infty)\,
Chi-squared f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{\frac{k}{2}\!-\!1} \exp\left(-\frac{x}{2}\right) \ln 2\Gamma\left(\frac{k}{2}\right) - \left(1 - \frac{k}{2}\right)\psi\left(\frac{k}{2}\right) + \frac{k}{2} E(x)=k,\,E(\ln(x))=\psi\left(\frac{k}{2}\right)+\ln(2) [0,\infty)\,
Erlang f(x) = \frac{\lambda^k}{(k-1)!} x^{k-1} \exp(-\lambda x) (1-k)\psi(k) + \ln \frac{\Gamma(k)}{\lambda} + k E(x)=k/\lambda,\,E(\ln(x))=\psi(k)-\ln(\lambda) [0,\infty)\,
F f(x) = \frac{n_1^{\frac{n_1}{2}} n_2^{\frac{n_2}{2}}}{B(\frac{n_1}{2},\frac{n_2}{2})} \frac{x^{\frac{n_1}{2} - 1}}{(n_2 + n_1 x)^{\frac{n_1 + n2}{2}}} \ln \frac{n_1}{n_2} B\left(\frac{n_1}{2},\frac{n_2}{2}\right) + \left(1 - \frac{n_1}{2}\right) \psi\left(\frac{n_1}{2}\right) -
\left(1 + \frac{n_2}{2}\right)\psi\left(\frac{n_2}{2}\right) + \frac{n_1 + n_2}{2} \psi\left(\frac{n_1\!+\!n_2}{2}\right)
\, [0,\infty)\,
Gamma f(x) = \frac{x^{k - 1} \exp(-\frac{x}{\theta})}{\theta^k \Gamma(k)} \ln(\theta \Gamma(k)) + (1 - k)\psi(k) + k \, E(x)=k\theta,\,E(\ln(x))=\psi(k)+\ln(\theta) [0,\infty)\,
Laplace f(x) = \frac{1}{2b} \exp\left(-\frac{|x - \mu|}{b}\right) 1 + \ln(2b) \, E(|x-\mu|)=b\, (-\infty,\infty)\,
Logistic f(x) = \frac{e^{-x}}{(1 + e^{-x})^2} 2 \, \, (-\infty,\infty)\,
Lognormal f(x) = \frac{1}{\sigma x \sqrt{2\pi}} \exp\left(-\frac{(\ln x - \mu)^2}{2\sigma^2}\right) \mu + \frac{1}{2} \ln(2\pi e \sigma^2) E(\ln(x))=\mu,E((\ln(x) - \mu)^2)=\sigma^2\, [0,\infty)\,
Maxwell-Boltzmann f(x) = \frac{1}{a^3}\sqrt{\frac{2}{\pi}}\,x^{2}\exp\left(-\frac{x^2}{2a^2}\right) \frac{1}{2}-\gamma_E-\ln(a\sqrt{2\pi}) E(x^2)=3a^2,\,E(\ln(x))\!=\!1\!+\!\ln\left(\frac{a}{\sqrt{2}}\right)\!-\!\frac{\gamma_E}{2} [0,\infty)\,
Generalized normal f(x) = \frac{2 \beta^{\frac{\alpha}{2}}}{\Gamma(\frac{\alpha}{2})} x^{\alpha - 1} \exp(-\beta x^2) \ln{\frac{\Gamma(\alpha/2)}{2\beta^{\frac{1}{2}}}} - \frac{\alpha - 1}{2} \psi\left(\frac{\alpha}{2}\right) + \frac{\alpha}{2} \, (-\infty,\infty)\,
Pareto f(x) = \frac{\alpha x_m^\alpha}{x^{\alpha+1}} \ln \frac{x_m}{\alpha} + 1 + \frac{1}{\alpha} E(\ln(x))=\frac{1}{\alpha}+\ln(x_m)\, [x_m,\infty)\,
Student's t f(x) = \frac{(1 + x^2/\nu)^{-\frac{\nu+1}{2}}}{\sqrt{\nu}B(\frac{1}{2},\frac{\nu}{2})} \frac{\nu\!+\!1}{2}\psi\left(\frac{\nu\!+\!1}{2}\right)\!-\!\psi\left(\frac{\nu}{2}\right)\!+\!\ln \sqrt{\nu} B\left(\frac{1}{2},\frac{\nu}{2}\right) E(\ln(x^2\!+\!\nu))=\log \left(\nu\right)\!-\!\psi \left(\frac{\nu}{2}\right)\!+\!\psi\left(\frac{\nu\!+\!1}{2} \right)\, (-\infty,\infty)\,
Triangular  f(x) = \begin{cases} \frac{2x}{a} & 0 \leq x \leq a\\ \frac{2(1-x)}{1-a} & a \leq x \leq 1 \end{cases} \frac{1}{2} - \ln 2 \, [0,1]\,
Weibull f(x) = \frac{k}{\lambda^k} x^{k-1} \exp\left(-\frac{x^k}{\lambda^k}\right) \frac{(k-1)\gamma_E}{k} + \ln \frac{\lambda}{k} + 1 E(x^k)=\lambda^k,E(\ln(x))=\ln(\lambda)-\frac{\gamma_E}{k}\, [0,\infty)\,
Multivariate normal 
f_X(\vec{x}) =
 \frac{\exp \left( -\frac{1}{2} ( \vec{x} - \vec{\mu})^\top \Sigma^{-1}\cdot(\vec{x} - \vec{\mu}) \right)} {(2\pi)^{N/2} \left|\Sigma\right|^{1/2}}
\frac{1}{2}\ln\{(2\pi e)^{N} \det(\Sigma)\} \, (-\vec{\infty},\vec{\infty})\,

(Many of the differential entropies are from [4].


As described above, differential entropy does not share all properties of discrete entropy. A modification of differential entropy adds an invariant measure factor to correct this, (see limiting density of discrete points). If m(x) is further constrained to be a probability density, the resulting notion is called relative entropy in information theory:

D(p||m) = \int p(x)\log\frac{p(x)}{m(x)}\,dx.

The definition of differential entropy above can be obtained by partitioning the range of X into bins of length h with associated sample points ih within the bins, for X Riemann integrable. This gives a quantized version of X, defined by Xh = ih if  ih \leq X \leq (i+1)h. Then the entropy of Xh is

H_h=-\sum_i hf(ih)\log f(ih) - \sum hf(ih)\log h.

The first term on the right approximates the differential entropy, while the second term is approximately − log(h). Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable should be \infty.

See also


  1. ^ Kraskov, Alexander; Stögbauer, Grassberger (2004). "Estimating mutual information". Phys. Rev. E 60: 066138. arXiv:cond-mat/0305641. Bibcode 2004PhRvE..69f6138K. doi:10.1103/PhysRevE.69.066138. 
  2. ^ Fazlollah M. Reza (1961, 1994). An Introduction to Information Theory. Dover Publications, Inc., New York. ISBN 0-486-68210-2. http://books.google.com/books?id=RtzpRAiX6OgC&pg=PA8&dq=intitle:%22An+Introduction+to+Information+Theory%22++%22entropy+of+a+simple+source%22&as_brr=0&ei=zP79Ro7UBovqoQK4g_nCCw&sig=j3lPgyYrC3-bvn1Td42TZgTzj0Q. 
  3. ^ Park, Sung Y.; Bera, Anil K. (2009). "Maximum entropy autoregressive conditional heteroskedasticity model". Journal of Econometrics (Elsevier): 219–230. http://www.wise.xmu.edu.cn/Master/Download/..%5C..%5CUploadFiles%5Cpaper-masterdownload%5C2009519932327055475115776.pdf. Retrieved 2011-06-02. 
  4. ^ Lazo, A. and P. Rathie. On the entropy of continuous probability distributions Information Theory, IEEE Transactions on, 1978. 24(1): p. 120-122.
  • Thomas M. Cover, Joy A. Thomas. Elements of Information Theory New York: Wiley, 1991. ISBN 0-471-06259-6

External links

Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Entropy (information theory) — In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information… …   Wikipedia

  • Entropy in thermodynamics and information theory — There are close parallels between the mathematical expressions for the thermodynamic entropy, usually denoted by S , of a physical system in the statistical thermodynamics established by Ludwig Boltzmann and J. Willard Gibbs in the 1870s; and the …   Wikipedia

  • Entropy estimation — Estimating the differential entropy of a system or process, given some observations, is useful in various science/engineering applications, such as Independent Component Analysis [Dinh Tuan Pham (2004) Fast algorithms for mutual information based …   Wikipedia

  • Entropy (arrow of time) — Entropy is the only quantity in the physical sciences that picks a particular direction for time, sometimes called an arrow of time. As one goes forward in time, the second law of thermodynamics says that the entropy of an isolated system can… …   Wikipedia

  • Differential pulse-code modulation — (DPCM) is a signal encoder that uses the baseline of pulse code modulation (PCM) but adds some functionalities based on the prediction of the samples of the signal. The input can be an analog signal or a digital signal. If the input is a… …   Wikipedia

  • Entropy — This article is about entropy in thermodynamics. For entropy in information theory, see Entropy (information theory). For a comparison of entropy in information theory with entropy in thermodynamics, see Entropy in thermodynamics and information… …   Wikipedia

  • Entropy (order and disorder) — Boltzmann s molecules (1896) shown at a rest position in a solid In thermodynamics, entropy is commonly associated with the amount of order, disorder, and/or chaos in a thermodynamic system. This stems from Rudolf Clausius 1862 assertion that any …   Wikipedia

  • Entropy and life — Much writing has been devoted to Entropy and life. Research concerning the relationship between the thermodynamic quantity entropy and the evolution of life began in around the turn of the 20th century. In 1910, American historian Henry Adams… …   Wikipedia

  • Principle of maximum entropy — This article is about the probability theoretic principle. For the classifier in machine learning, see maximum entropy classifier. For other uses, see maximum entropy (disambiguation). Bayesian statistics Theory Bayesian probability Probability… …   Wikipedia

  • Maximum entropy thermodynamics — In physics, maximum entropy thermodynamics (colloquially, MaxEnt thermodynamics) views equilibrium thermodynamics and statistical mechanics as inference processes. More specifically, MaxEnt applies inference techniques rooted in Shannon… …   Wikipedia