Do Differentiable Simulators Give Better Policy Gradients?

Authors: H.J. Terry Suh, Max Simchowitz, Kaiqing Zhang, Russ Tedrake · ICML 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Exact simulator gradients can be biased or high-variance under contact, stiffness and chaos; an α-order estimator blends them with sampled gradients under an accuracy bound.

Why it matters

The paper presents its minimal examples (friction, stiff contact, collisions with sharp geometry) as abstractions of contact-rich tasks such as robotic manipulation, and argues that such tasks necessarily involve these characteristics. Its findings suggest that, for policies trained with a differentiable simulator, exact gradients can steer optimization to worse minima in these cases, and that simulator modelling choices such as contact stiffness affect policy optimization; the authors offer this as a possible explanation for why zeroth-order RL has solved physical-system problems where first-order methods struggled, but they test it only on simple simulated systems, not robots.

Problem

Differentiable simulators give exact gradients of a rollout, which can replace the zeroth-order (sampling-based) gradient estimates that reinforcement learning usually relies on. It was unclear which estimator works better on the long-horizon, non-smooth objectives that arise in control of physical systems with contact.

Proposed solution

The authors analyse the first-order batched gradient (FoBG, averaged exact gradients) and the zeroth-order batched gradient (ZoBG, REINFORCE-style) on a noise-smoothed policy objective, through bias and variance. They then propose the α-order batched gradient (AoBG), a weighted mix of the two that picks α to minimise variance subject to a constraint that keeps it close to the unbiased zeroth-order estimate, falling back to α = 0 when there are too few samples to guarantee accuracy.

Key insights

Under discontinuities such as Coulomb friction or sharp collision geometry, the first-order estimator is biased while the zeroth-order estimator stays unbiased.
Stiff continuous approximations of discontinuities cause 'empirical bias': with finite samples the first-order estimate shows low empirical variance yet is far from the true gradient, so empirical variance alone cannot detect the problem.
Even without bias, first-order estimates can have higher variance than zeroth-order ones under stiff contact or chaotic dynamics, because the variance of the zeroth-order estimate scales with horizon times state dimension but not with the size of the derivatives.
The two estimators' bias and variance lead them not just to different convergence rates but to different local minima.
Interpolating by minimum empirical variance, as in earlier work, can be badly inaccurate near discontinuities; adding an accuracy constraint fixes this.

Main results

For a piecewise-linear relaxation of Coulomb friction with slip tolerance ν, the first-order estimate has (c_σν, c_σ, 0) empirical bias and its variance scales with 1/ν as ν → 0 (with c_σ = 1/(√(2π)σ)); at ν = 0 it becomes biased in expectation with zero variance.
The variance of the zeroth-order estimate is bounded by B_V²B_π²·Hn/(Nσ²), scaling with horizon H and state dimension n but not with derivative magnitude (Lemma 3.10).
Block pushing with penalty-method contact, horizon H = 200: the first-order estimator outperformed the zeroth-order one with soft springs (k = 10), the zeroth-order one won with stiff springs (k = 1000), and the α-order estimator matched the better of the two in each case.
Tennis (breakout-like paddle) with a linear feedback policy of d = 21 parameters and H = 200: zeroth-order and α-order estimators found policies that bounced the balls from different initial conditions, the first-order estimator still missed many balls, and the α-order estimator converged slightly faster than the zeroth-order one.
On the ball-with-wall and momentum-transfer examples, the first-order estimator stalled in a flat region or was pushed off the cliff into a high-cost region, while the zeroth-order and α-order estimators reached the better minima.
On the double pendulum, the first-order estimator's variance exceeded the zeroth-order one's as the horizon grew.

Limitations

Simulators based on optimization-based implicit time-stepping (such as MuJoCo, PyBullet and quasistatic schemes) were not studied empirically; the authors expect geometric discontinuities to remain a problem there and leave this to future work.
All demonstrations are simple numerical examples written in PyTorch (ball, wall, pivot bar, pushed blocks, sliding box, double pendulum, paddle); no robot, robot model or robot-learning benchmark is used.
The stiff-contact variance issue holds only until semi-implicit integration becomes unstable at a fixed timestep; lowering the timestep helps at the cost of computation time.
The accuracy constraint needs a prior upper bound on the value function and on the policy Jacobian, and uses independent trajectories for the two estimators; when samples are too few to meet the bound, the method falls back to the zeroth-order estimate.
The analysis assumes Gaussian injected noise on the actions, which is treated as smoothing of the objective.

Field impact

Winner of the ICML 2022 Outstanding Paper Award, and cited 166 times. The paper does not state a release of code or data.

Abstract

Differentiable simulators promise faster computation time for reinforcement learning by replacing zeroth-order gradient estimates of a stochastic objective with an estimate based on first-order gradients. However, it is yet unclear what factors decide the performance of the two estimators on complex landscapes that involve long-horizon planning and control on physical systems, despite the crucial relevance of this question for the utility of differentiable simulators. We show that characteristics of certain physical systems, such as stiffness or discontinuities, may compromise the efficacy of the first-order estimator, and analyze this phenomenon through the lens of bias and variance. We additionally propose an α-order gradient estimator, with α ∈ [0, 1], which correctly utilizes exact gradients to combine the efficiency of first-order estimates with the robustness of zero-order methods. We demonstrate the pitfalls of traditional estimators and the advantages of the α-order estimator on some numerical examples.

Open Source Robotics