Hamilton-Jacobi-Bellman equation

The Hamilton-Jacobi-Bellman (HJB) equation is a partial differential equation which is central to optimal control theory.

The solution of the HJB equation is the 'value function', which gives the optimal cost-to-go for a given dynamical system with an associated cost function. Classical variational problems, for example, the brachistochrone problem can be solved using this method as well.

The equation is a result of the theory of dynamic programming which was pioneered in the 1950s by Richard Bellman and coworkers. [R. E. Bellman. Dynamic Programming. Princeton, NJ, 1957.] The corresponding discrete-time equation is usually referred to as the Bellman equation. In continuous time, the result can be seen as an extension of earlier work in classical physics on the Hamilton-Jacobi equation by William Rowan Hamilton and Carl Gustav Jacob Jacobi.

Consider the following problem in deterministic optimal control

:$min\; int\_0^T\; C\; [x(t),u(t)]\; ,dt\; +\; D\; [x(T)]$

subject to

:$dot\{x\}(t)=F\; [x(t),u(t)]$

where $x(t)$ is the system state, $x(0)$ is assumed given, and $u(t)$ for $0leq\; tleq\; T$ is the control that we are trying to find.For this simple system, the Hamilton Jacobi Bellman partial differential equation is

:$frac\{partial\}\{partial\; t\}\; V(x,t)\; +\; min\_u\; left\{\; leftlangle\; frac\{partial\}\{partial\; x\}V(x,t),\; F(x,\; u)\; ight\; angle\; +\; C(x,u)\; ight\}\; =\; 0$

subject to the terminal condition

:$V(x,T)\; =\; D(x).,$

The unknown $V(t,\; x)$ in the above PDE is the Bellman 'value function', which represents the cost incurred from starting in state $x$ at time $t$ and controlling the system optimally from then until time $T$.The HJB equation needs to be solved backwards in time, starting from $t\; =\; T$ and ending at $t\; =\; 0$. (The notation $langle\; a,b\; angle$ means the inner product of the vectors a and b).

The HJB equation is a sufficient condition for an optimum.Fact|date=May 2008 If we can solve for $V$ then we can find from it a control $u$ that achieves the minimum cost.

The HJB method can be generalized to stochastic systems as well.

In general case, the HJB equation does not have a classical (smooth) solution. Several notions of generalized solutions have been developed to cover such situations, including viscosity solution (Pierre-Louis Lions and Michael Crandall), minimax solution (Andrei Izmailovich Subbotin), and others.

References