Differential dynamic programming
- Also called: DDP
- Category: Planning & manipulation › Trajectory optimisation
- Tags: trajectory-optimization, optimal-control, model-predictive-control, numerical-optimization, shooting-methods
A trajectory optimization method that improves a control sequence by repeated backward and forward passes, using local quadratic models of the cost-to-go along the current trajectory.
Definition
Differential dynamic programming (DDP) is an algorithm for trajectory optimization: given a dynamics model, a cost over states and controls, and an initial state, it finds a control sequence that locally minimizes the total cost. It was introduced by David Mayne in 1966 and developed further in Jacobson and Mayne's 1970 book of the same name. DDP is a shooting method: the decision variables are the controls, and in its classical form the states are always obtained by simulating the dynamics forward, so every iterate is a dynamically feasible trajectory.
Each iteration has two passes. The backward pass starts at the final timestep and works toward the start, applying Bellman's principle of optimality locally: it builds a quadratic approximation of the cost-to-go around the current trajectory from first and second derivatives of the dynamics and cost, and solves for an affine control update at every step, made of a feedforward term k and a feedback gain matrix K. The forward pass then simulates the system with the updated controls u = ū + αk + K(x − x̄), where α is a line-search step size. Because the expansion includes second-order terms of the dynamics, DDP converges quadratically near a local minimum, like Newton's method, while the work per iteration grows only linearly with the horizon length.
The second derivatives of the dynamics are expensive to compute, so the most widely used variant, the iterative linear quadratic regulator (iLQR), drops them and keeps a first-order model of the dynamics, a Gauss-Newton approximation. Practical solvers regularize the backward pass to keep it well conditioned and extend the method to control limits (control-limited DDP), general constraints (augmented Lagrangian methods such as ALTRO), infeasible initial guesses (feasibility-driven DDP in Crocoddyl) and, as in Parameterized DDP, time-invariant parameters optimized together with the controls. DDP finds local optima only, so results depend on the initial guess. A useful by-product is the set of feedback gains, which can stabilize the robot around the optimized trajectory.
Why it matters
DDP and iLQR are among the most used solvers for robot trajectory optimization and nonlinear model predictive control. They exploit the time structure of the problem, scale linearly with horizon length and return a local feedback policy along with the trajectory, which is why legged robots, humanoids and drones run them in real time on board.
Key applications
- Nonlinear model predictive control for legged robots and humanoids
- Trajectory optimization for drones and agile vehicles
- Torque-level motion planning for manipulators
- Swing-up and balance of underactuated systems
- Joint optimization of trajectories and model or design parameters
In practice
A team building whole-body model predictive control for a quadruped formulates the problem in Crocoddyl. The contact sequence comes from a fixed gait schedule, so each phase has known contacts, and the cost combines base tracking, foot placement, control effort and penalties that keep contact forces inside friction cones. Every control cycle, the solver is warm-started with the previous solution shifted by one step and runs only one or a few iterations. The first control and its feedback gain are sent to the low-level joint controller, which uses the gain to correct for state changes between solver updates. Most of the tuning effort goes into cost weights, regularization and the horizon length, which trade foresight against solve time.
Common misconceptions
- DDP finds the globally optimal trajectory — it is a local method; the result depends on the initial guess and can be a poor local minimum
- iLQR and DDP are the same algorithm — iLQR omits the second-order dynamics terms, giving much cheaper iterations at the cost of slower convergence near the solution
- DDP cannot handle constraints — the basic algorithm is unconstrained, but control-limited and augmented Lagrangian variants are widely used
- DDP is dynamic programming over the whole state space — it applies Bellman's recursion only along one trajectory, which avoids the curse of dimensionality but gives only local results
Who should know this
- Control engineers building model predictive controllers for legged robots and drones
- Trajectory optimization researchers
- Humanoid and whole-body control developers
- Robotics students studying optimal control
Related technologies
- iLQR
- Crocoddyl
- ALTRO
- Riccati Recursion
- Shooting Methods
Related terms
- Direct Collocation
- Model Predictive Control
- Trajectory Planning
- Moving horizon estimation
- Underactuated system
- Hybrid system
Related projects
- PyRoki: Modular Python toolkit for kinematic optimization across CPU, GPU, and TPU via JAX
- MuJoCo MPC: DeepMind's interactive real-time model predictive control framework for MuJoCo
- python_simple_mppi: MPPI control derived and implemented in numpy, with four worked examples
- mjbatch: Runs thousands of MuJoCo simulations in parallel on the CPU
- Quadruped-PyMPC: Python MPC framework for quadruped locomotion with gradient and sampling control
- STLCG: JAX toolbox for differentiable Signal Temporal Logic robustness