An Online Learning Approach to Model Predictive Control

Authors: Nolan Wagener, Ching-An Cheng, Jacob Sacks, Byron Boots · RSS 2019

Licences

Open Access
Paper: CC0 1.0 Universal

Everything below draws on the full paper, which its authors released under CC0 1.0 Universal; its figure is credited to them.

TL;DR

Viewing MPC as online learning gives DMD-MPC, a family that contains MPPI and CEM, exposes their step size as a tuning knob, and runs on a 1:5 scale AutoRally car.

Why it matters

The framework turns MPC design into choosing a loss, a control distribution and a divergence, and makes the step size an explicit parameter. In the simulated AutoRally sweeps, a smaller step size let the controller keep most of its performance with far fewer samples, which the authors point to for embedded systems that cannot draw many rollouts.

Problem

MPC algorithms such as MPPI and the cross-entropy method were derived from different assumptions and heuristics, so there is no common recipe for designing or tuning them. Their update rules fix design choices, such as a step size of 1, that are not necessarily optimal, which matters when computation limits how many rollouts can be sampled.

Proposed solution

The authors map MPC onto online learning: each control step is a round, the decision is the parameter of a control distribution, and the per-round loss is the MPC objective at the current state. Applying dynamic mirror descent, with the MPC shift operator as its predictive model, gives DMD-MPC, a family defined by three choices: the loss (expected cost, probability of low cost, or exponential utility), the control distribution (e.g., Gaussian, categorical, Laplace) and the Bregman divergence. MPPI and CEM are recovered as special cases with step size 1.

Key insights

Warm-starting MPC from the shifted previous solution plays the role of the predictive model in dynamic mirror descent, and dynamic regret measures how consistent the plan stays with the MPC objective.
MPPI follows from the exponential-utility loss and CEM from the probability-of-low-cost loss, both with step size 1; the step size is a free parameter that the original algorithms fix.
For exponential-family control distributions, the update is a convex combination of the previous expectation parameter and the utility-weighted sufficient statistics.
Utility-based losses give near-zero weight to high-cost rollouts; with the plain expected-cost loss, rollouts that leave the track dominate the gradient estimate and the car drove slowly or hit walls.
Discrete controls are handled with a categorical distribution and an exponentiated-gradient update, and the cost can be discontinuous, such as an indicator function.

Main results

Cartpole in simulation, with a biased model (0.346 m pole in the model vs 0.326 m true): with continuous controls, step sizes above 1 gave significant gains over the step size of 1 used in MPPI and CEM across a large range of utility parameters.
Simulated AutoRally: gains were small beyond 1920 samples per gradient, and with a proper step size 192 samples gave lap times within a couple of seconds of 3840 samples at step size 1.
Simulated AutoRally with the expected-cost loss: performance was dramatically worse even with 3840 samples; at best the car stayed in the track centre below 4 m/s.
Real AutoRally, 1920 samples: lap times of 9.71 ± 0.17 s (Gaussian, step 1, i.e., MPPI), 9.76 ± 0.14 s (Gaussian, step 0.9) and 10.03 ± 0.16 s (Laplace, step 0.9) at a 13 m/s target; 10.49 ± 0.08, 10.66 ± 0.09 and 10.91 ± 0.14 s at 7 m/s.
Real AutoRally, 12 m/s target, second set: with 1920 samples, step 0.8 gave 10.70 ± 0.19 s vs 10.90 ± 0.16 s at step 1; with 192 samples, steps 0.8 and 0.6 both gave 10.97 s vs 11.10 ± 0.17 s at step 1.

Limitations

Small dynamic regret does not imply good absolute control performance; it only means the plan is consistent with the chosen MPC objective.
The two real-world experiment sets ran under different conditions (the second on a poor track that limited top speed) and cannot be compared with each other; 1920 samples at step size 0.6 was not tested for lack of time.
With few samples and a large step size (64 samples, step 1) the simulated car drove slowly and erratically, sometimes stopping.
On the real car the new variants performed comparably to MPPI rather than better: in the first set MPPI (Gaussian, step 1) had the lowest lap time at both speed targets.
The AutoRally dynamics model is a deterministic neural network fitted to human driving data, and only the mean of the control distribution was optimized in the real-world runs.

Field impact

Won the RSS 2019 Best Student Paper Award and was a finalist for the RSS 2019 Best Systems Paper Award; it has 86 citations. The paper does not link code or data.

Abstract

Model predictive control (MPC) is a powerful technique for solving dynamic control tasks. In this paper, we show that there exists a close connection between MPC and online learning, an abstract theoretical framework for analyzing online decision making in the optimization literature. This new perspective provides a foundation for leveraging powerful online learning algorithms to design MPC algorithms. Specifically, we propose a new algorithm based on dynamic mirror descent (DMD), an online learning algorithm that is designed for non-stationary setups. Our algorithm, Dynamic Mirror Decent Model Predictive Control (DMD-MPC), represents a general family of MPC algorithms that includes many existing techniques as special instances. DMD-MPC also provides a fresh perspective on previous heuristics used in MPC and suggests a principled way to design new MPC algorithms. In the experimental section of this paper, we demonstrate the flexibility of DMD-MPC, presenting a set of new MPC algorithms on a simple simulated cartpole and a simulated and real-world aggressive driving task.

Open Source Robotics