Observed information


Observed information

In statistics, the observed information, or observed Fisher information, is the negative of the second derivative (the Hessian matrix) of the "log-likelihood" (the logarithm of the likelihood function). It is a sample-based version of the Fisher information.

Contents

Definition

Suppose we observe random variables X_1,\ldots,X_n, independent and identically distributed with density f(X; θ), where θ is a (possibly unknown) vector. Then the log-likelihood of the parameters θ given the data X_1,\ldots,X_n is

\ell(\theta | X_1,\ldots,X_n) = \sum_{i=1}^n \log f(X_i| \theta) .

We define the observed information matrix at θ * as

\mathcal{J}(\theta^*) 
  = - \left. 
    \nabla \nabla^{\top} 
    \ell(\theta)
  \right|_{\theta=\theta^*}
= -
\left.
\left( \begin{array}{cccc}
  \tfrac{\partial^2}{\partial \theta_1^2}
  &  \tfrac{\partial^2}{\partial \theta_1 \partial \theta_2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_1 \partial \theta_n} \\
  \tfrac{\partial^2}{\partial \theta_2 \partial \theta_1}
  &  \tfrac{\partial^2}{\partial \theta_2^2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_2 \partial \theta_n} \\
  \vdots &
  \vdots &
  \ddots &
  \vdots \\
  \tfrac{\partial^2}{\partial \theta_n \partial \theta_1}
  &  \tfrac{\partial^2}{\partial \theta_n \partial \theta_2}
  &  \cdots
  &  \tfrac{\partial^2}{\partial \theta_n^2} \\
\end{array} \right) 
\ell(\theta)
\right|_{\theta = \theta^*}

In many instances, the observed information is evaluated at the maximum-likelihood estimate.[1]

Fisher information

The Fisher information \mathcal{I}(\theta) is the expected value of the observed information:

\mathcal{I}(\theta) = \mathrm{E}(\mathcal{J}(\theta)).

Applications

In a notable article, Bradley Efron and David V. Hinkley [2] argued that the observed information should be used in preference to the expected information when employing normal approximations for the distribution of maximum-likelihood estimates.

References

  1. ^ Dodge, Y. (2003) The Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9
  2. ^ Efron, B.; Hinkley, D.V. (1978). "Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher Information". Biometrika 65 (3): 457–487. doi:10.1093/biomet/65.3.457. JSTOR 2335893. MR0521817. 

Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Information needs — Information need is an individual or group s desire to locate and obtain information to satisfy a conscious or unconscious need. The ‘information’ and ‘need’ in ‘information need’ are inseparable interconnection. Needs and interests call forth… …   Wikipedia

  • Information economy — is a defined term that an economy with an increased emphasis on informational activities and information industry. The vagueness of the term has three major sources. First, not surprisingly, there is no agreed upon definition regarding the… …   Wikipedia

  • Information processor — An information processor or information processing system, as its name suggests, is a system (be it electrical, mechanical or biological) which takes information (a sequence of enumerated states) in one form and processes (transforms) it into… …   Wikipedia

  • information theory — the mathematical theory concerned with the content, transmission, storage, and retrieval of information, usually in the form of messages or data, and esp. by means of computers. [1945 50] * * * ▪ mathematics Introduction       a mathematical… …   Universalium

  • Information and communication technologies for development — An OLPC class in Ulaanbaatar, Mongolia …   Wikipedia

  • information foraging — (in.fuhr.may.shuhn FOR.uh.jing) n. Searching for information, especially by using strategies analogous to the food foraging techniques employed by animals. information forager n. information forage v. Example Citation: Information foraging theory …   New words

  • Information bottleneck method — The information bottleneck method is a technique introduced by Tishby et al [1] for finding the best tradeoff between accuracy and complexity (compression) when summarizing (e.g. clustering) a random variable X, given a joint probability… …   Wikipedia

  • Information set (game theory) — In game theory, an information set is a set that, for a particular player, establishes all the possible moves that could have taken place in the game so far, given what that player has observed so far. If the game has perfect information, every… …   Wikipedia

  • traffic information (radar) — Information issued to alert an aircraft to any radar targets observed on the radar display that may be in such proximity to its position or intended route of flight to warrant its attention …   Military dictionary

  • traffic information — Information issued by an air traffic services unit to alert a pilot to other known or observed air traffic which may be in proximity to the position or intended route of flight and to help the pilot to avoid a collision (ICAO) …   Aviation dictionary