Hamilton-Jacobi-Bellman Equation (HJB)

• Also called: HJB, HJB equation
• Category: Control › Feedback control
• Tags: optimal-control, dynamic-programming, value-function, pde, safety

The partial differential equation whose solution is the optimal value function of a continuous-time optimal control problem; the continuous-time form of Bellman's principle of optimality.

Definition

The Hamilton-Jacobi-Bellman (HJB) equation is the continuous-time counterpart of the Bellman equation. For a system with dynamics ẋ = f(x, u), running cost ℓ(x, u) and terminal cost φ(x), the optimal cost-to-go V(x, t) satisfies −∂V/∂t = min over u of [ ℓ(x, u) + ∇V(x, t) · f(x, u) ], with V(x, T) = φ(x) at the final time. The equation follows from Bellman's principle of optimality: the best cost from any state equals the cost of the best first infinitesimal step plus the optimal cost from where that step leads. Once V is known, the optimal control at any state is the u that achieves the minimum, so the solution gives a feedback policy over the whole state space, not just one trajectory.

For stochastic systems driven by noise, dx = f(x, u) dt + σ(x) dW, the equation gains a second-order term, ½ tr(σσᵀ ∇²V), that accounts for diffusion. Infinite-horizon and discounted problems give a stationary form without the time derivative. For linear dynamics with quadratic cost, V is quadratic and the HJB equation reduces to the Riccati equation behind the linear-quadratic regulator. In general V need not be differentiable, and the correct notion of solution is the viscosity solution introduced by Crandall and Lions in 1983. Pontryagin's maximum principle is the complementary approach: it gives necessary conditions along a single optimal trajectory rather than a value function over all states.

The main obstacle is the curse of dimensionality: grid-based solvers scale exponentially with the state dimension, so exact solutions are practical only for systems with a handful of states. Many robotics methods can be read as ways around this. Differential dynamic programming and iLQR expand the value function locally along a trajectory; path-integral control uses a transformation that makes the stochastic HJB equation linear; deep learning methods approximate V with neural networks, for example by solving equivalent forward-backward stochastic differential equations. A closely related equation, the Hamilton-Jacobi-Isaacs equation, describes two-player games and underlies Hamilton-Jacobi reachability, which computes sets of states from which a system can be kept safe despite disturbances.

Why it matters

The HJB equation is the theoretical reference point for optimal feedback control in continuous time. It connects LQR, trajectory optimization, path-integral methods, reachability-based safety and continuous-time reinforcement learning, and it explains why exact optimal control is limited to low-dimensional systems.

Key applications

• Deriving optimal feedback controllers, including LQR via the Riccati equation
• Hamilton-Jacobi reachability for safety verification and safe sets
• Path-integral and sampling-based stochastic optimal control
• Continuous-time reinforcement learning and value-function approximation
• Neural and forward-backward SDE solvers for high-dimensional optimal control

In practice

A team designing a safety controller for a small ground robot models it as a three-state Dubins-style vehicle with a bounded disturbance. They solve the related Hamilton-Jacobi reachability equation on a grid over position and heading with a level-set toolbox, which gives a value function whose zero level set marks the states from which a collision can always be avoided. Offline, the computation takes minutes; online, the robot looks up the value and its gradient at the current state to decide when to override the nominal planner. For a manipulator with many more states, the same grid approach would be infeasible, so the team falls back to local methods such as iLQR or learned value approximations.

Common misconceptions

• The HJB equation is only of theoretical interest — it is solved directly for low-dimensional systems in reachability analysis and underlies practical methods such as LQR, DDP and MPPI
• The value function is always smooth — in many problems V has kinks, and the equation must be understood in the viscosity sense
• The HJB equation and Pontryagin's maximum principle give the same thing — HJB gives a value function and feedback law over all states, while Pontryagin gives necessary conditions along one trajectory

Who should know this

• Control engineers working on optimal and safe control
• Researchers in reachability analysis and safety verification
• Reinforcement learning researchers working in continuous time
• Graduate students in control theory and robotics

Related technologies

• Linear-Quadratic Regulator (LQR)
• Riccati Equation
• Pontryagin's Maximum Principle
• Level Set Methods
• Hamilton-Jacobi Reachability
• Forward-Backward Stochastic Differential Equations

Related terms

• Markov Decision Process (MDP)
• Differential dynamic programming
• Model Predictive Path Integral
• Control Barrier Function
• Safety Filter
• Region of attraction

Related projects

• NASA Ogma: NASA runtime monitor generator for safe robotics and flight systems

Videos

• Nonlinear Control: Hamilton Jacobi Bellman (HJB) and Dynamic Programming: Steve Brunton