Isotonic regression

Isotonic regression

In numerical analysis, isotonic regression (IR) involves finding a weighted least-squares fit x\in \Bbb{R}^n to a vector a\in \Bbb{R}^n with weights vector w\in \Bbb{R}^n subject to a set of monotonicity constraints giving a simple or partial order over the variables. The monotonicity constraints define a directed acyclic graph G = (C,P) over the nodes N={1,2,\ldots,n} corresponding to the variables x={x_1,x_2,\ldots,x_n}. Thus, the IR problem where a simple order is defined corresponds to the following quadratic program (QP):

\min \sum_{i=1}^n w_i (x_i - a_i)^2 \mathrm{subject~to~}x_i\ge x_j~\forall (i,j)\in E.

In the case when G = (N,E) is a total order, a simple iterative algorithm for solving this QP is called the pool adjacent violators algorithm (PAVA). Best and Chakravarti (1990) have studied the problem as an active set identification problem, and have proposed a primal algorithm in O(n), the same complexity as the PAVA, which can be seen as a dual algorithm.

IR has applications in statistical inference, for example, computing the cost at the minimum of the above goal function, gives the "stress" of the fit of an isotonic curve to mean experimental results when an order is expected.

Another application is nonmetric multidimensional scaling (Kruskal, 1964), where a low-dimensional embedding for data points is sought such that order of distances between points in the embedding matches order of dissimilarity between points. Isotonic regression is used iteratively to fit ideal distances to preserve relative dissimilarity order.

Isotonic regression is also sometimes referred to as monotonic regression. Correctly speaking, isotonic is used when the direction of the trend is strictly increasing, while monotonic could imply a trend that is either strictly increasing or strictly decreasing.

Isotonic Regression under the Lp for p > 0 is defined as follows:

\min \sum_{i=1}^n w_i |x_i - a_i|^p \mathrm{subject~to~}x_i\ge x_j~\forall (i,j)\in E.

Simply ordered case

To illustrate the above, let x_1 \leq x_2 \leq \ldots \leq x_n, and f(x_1) \leq f(x_2) \leq \ldots \leq f(x_n), and w_i \geq 0 .

The isotonic estimator, g * , minimizes the weighted least squares-like condition:

\min_g \sum_{i=1}^n w_i (g(x_i) - f(x_i))^2

Where g is the unknown function we are estimating, and f is a known function.

Software has been developed in the R statistical package for computing isotone (monotonic) regression.[1]


  1. ^ De Leeuw, Jan; K. Hornik, P. Mair (2009). "Isotone Optimization in R: Pool-Adjacent-Violators Algorithm (PAVA) and Active Set Methods". Journal of statistical software 32 (5): 1. 
  • Best, M.J.; & Chakravarti N. (1990). "Active set algorithms for isotonic regression; a unifying framework". Mathematical Programming 47: 425–439. doi:10.1007/BF01580873. 
  • Robertson, T.; Wright, F.T.; & Dykstra, R.L. (1988). Order restricted statistical inference. New York: Wiley, 1988. ISBN 0-471-91787-7
  • Barlow, R. E.; Bartholomew, D.J.; Bremner, J. M.; & Brunk, H. D. (1972). Statistical inference under order restrictions; the theory and application of isotonic regression. New York: Wiley, 1972. ISBN 0-471-04970-0.
  • Kruskal, J. B. (1964). "Nonmetric Multidimensional Scaling: A numerical method". Psychometrika 29 (2): 115–129. doi:10.1007/BF02289694. 
  • Shively, T.S., Sager, T.W., Walker, S.G. (2009). "A Bayesian approach to non-parametric monotone function estimation". J.R.Statist.Soc. B 71 (1): 159–175. doi:10.1111/j.1467-9868.2008.00677.x. 
  • Wu, W. B.; Woodroofe, M.; & Mentz, G. (2001). "Isotonic regression: Another look at the changepoint problem". Biometrika 88 (3): 793–804. doi:10.1093/biomet/88.3.793. 

Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Régression isotonique — En analyse numérique, la régression isotonique (« isotonic regression (IR) ») cherche à trouver un ajustement pondéré des moindres carrés à un vecteur avec des vecteurs pondérés sujets à un ensemble de contraintes de monotonie accordant …   Wikipédia en Français

  • Regression toward the mean — In statistics, regression toward the mean (also known as regression to the mean) is the phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on a second measurement, and a fact that may… …   Wikipedia

  • Regression discontinuity design — In statistics, econometrics, epidemiology and related disciplines, a regression discontinuity design (RDD) is a design that elicits the causal effects of interventions by exploiting a given exogenous threshold determining assignment to treatment …   Wikipedia

  • Nonparametric regression — is a form of regression analysis in which the predictor does not take a predetermined form but is constructed according to information derived from the data. Nonparametric regression requires larger sample sizes than regression based on… …   Wikipedia

  • Outline of regression analysis — In statistics, regression analysis includes any technique for learning about the relationship between one or more dependent variables Y and one or more independent variables X. The following outline is an overview and guide to the variety of… …   Wikipedia

  • Linear regression — Example of simple linear regression, which has one independent variable In statistics, linear regression is an approach to modeling the relationship between a scalar variable y and one or more explanatory variables denoted X. The case of one… …   Wikipedia

  • Robust regression — In robust statistics, robust regression is a form of regression analysis designed to circumvent some limitations of traditional parametric and non parametric methods. Regression analysis seeks to find the effect of one or more independent… …   Wikipedia

  • Nonlinear regression — See Michaelis Menten kinetics for details In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or… …   Wikipedia

  • List of statistics topics — Please add any Wikipedia articles related to statistics that are not already on this list.The Related changes link in the margin of this page (below search) leads to a list of the most recent changes to the articles listed below. To see the most… …   Wikipedia

  • List of numerical analysis topics — This is a list of numerical analysis topics, by Wikipedia page. Contents 1 General 2 Error 3 Elementary and special functions 4 Numerical linear algebra …   Wikipedia