Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions

Authors: Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, Pieter Abbeel · IROS 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A style reward learned adversarially from 4.5 s of German Shepherd motion capture replaces a hand-designed reward and transfers to a real A1 quadruped with a lower cost of transport.

Why it matters

Reward tuning is a large share of the engineering work in sim-to-real legged locomotion. The paper shows that a few seconds of reference motion can stand in for a 13-term hand-designed style reward on a real quadruped, while the policy still departs from the clips as needed to track commanded velocities and changes gait with speed.

Problem

Reinforcement learning agents for legged locomotion trained with an under-specified reward often learn behaviours such as flailing limbs, leg vibration or high-impulse contacts that exploit simulator inaccuracies and cannot run on a real robot. The usual remedy is a complex hand-designed style reward, which needs substantial domain knowledge and delicate tuning, and tends to be specific to one platform and task.

Proposed solution

The authors adapt Adversarial Motion Priors (AMP) from computer graphics: a discriminator trained with a least-squares GAN objective and a gradient penalty learns to tell state transitions in a motion-capture dataset from transitions produced by the policy, and its output becomes a style reward that is added to a velocity-tracking task reward (weights 0.65 and 0.35). German Shepherd clips totalling 4.5 seconds are retargeted to the A1 robot with inverse kinematics, and the policy (target joint angles at 30 Hz, tracked by PD controllers) is trained with PPO across 5,280 Isaac Gym environments with domain randomization before deployment on hardware.

Key insights

An adversarially learned style reward from 4.5 s of motion capture can replace a hand-designed style reward with 13 terms and still yield policies that can be deployed on a real quadruped.
Because the discriminator scores state transitions instead of tracking a specific clip, the policy can follow velocity commands that do not appear in the reference data.
Policies trained with the motion prior change gait with commanded speed (pace at 0.8 m/s, trot at 1.7 m/s, and a switch from pace to canter when the command rises from 1 m/s to 2 m/s), which the authors relate to how animals lower energy use.
With no style reward, the policy exploits inaccurate simulator dynamics by vibrating its feet at high speed and torque; it tracks velocity well but is too violent to run on the real robot.
The authors attribute the lower cost of transport to energy-efficient strategies already present in the dog's motion data.

Main results

Mechanical cost of transport in simulation (Table II), at commanded speeds of 0.4, 0.8, 1.2 and 1.6 m/s: 1.07, 0.93, 1.02 and 1.12 with the AMP reward versus 1.54, 1.37, 1.40 and 1.41 with the complex style reward. The text gives the complex-reward range as 1.37 to 1.65, which does not match the table's highest value of 1.54.
With no style reward, cost of transport was 14.03, 8.00, 6.05 and 5.18 at the same four speeds, caused by high torques and motor velocities.
Measured forward velocity for commands of 0.4/0.8/1.2/1.6 m/s: 0.36/0.77/1.11/1.52 m/s with AMP, 0.41/0.88/1.28/1.67 m/s with the complex reward and 0.42/0.82/1.22/1.61 m/s with no style reward.
Training took 4 billion environment steps (about 4.2 years of simulated data) in 16 hours on a single Tesla V100 GPU.
On the real A1, the AMP policy tracked sinusoidal linear and angular velocity commands, steered through a route with sharp turns, and switched from pace to canter when the command rose from 1 m/s to 2 m/s.

Limitations

The paper has no limitations section. Its real-robot evaluation is qualitative (gait diagrams, velocity traces, cost-of-transport plots and a route-following example), with no success rates or energy tables measured on hardware.
All cost-of-transport and velocity-tracking numbers in Table II come from simulation.
The reference data is 4.5 s of a single German Shepherd (pacing, trotting, cantering and turning in place), retargeted to one robot, the A1; no other robot or skill is tested.
In simulation the AMP policy runs slightly below the commanded forward speed at every tested command (for example 1.52 m/s for a 1.6 m/s command).

Field impact

Finalist for the IROS 2022 Best Paper Award and ABB Best Student Paper Award, and cited 210 times. The paper points to a project page with videos and the project repository, which links the code at github.com/Alescontrela/AMP_for_hardware.

Abstract

Training a high-dimensional simulated agent with an under-specified reward function often leads the agent to learn physically infeasible strategies that are ineffective when deployed in the real world. To mitigate these unnatural behaviors, reinforcement learning practitioners often utilize complex reward functions that encourage physically plausible behaviors. However, a tedious labor-intensive tuning process is often required to create hand-designed rewards which might not easily generalize across platforms and tasks. We propose substituting complex reward functions with "style rewards" learned from a dataset of motion capture demonstrations. A learned style reward can be combined with an arbitrary task reward to train policies that perform tasks using naturalistic strategies. These natural strategies can also facilitate transfer to the real world. We build upon Adversarial Motion Priors – an approach from the computer graphics domain that encodes a style reward from a dataset of reference motions – to demonstrate that an adversarial approach to training policies can produce behaviors that transfer to a real quadrupedal robot without requiring complex reward functions. We also demonstrate that an effective style reward can be learned from a few seconds of motion capture data gathered from a German Shepherd and leads to energy-efficient locomotion strategies with natural gait transitions.

Open Source Robotics