Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

Authors: Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, Gavriel State · arXiv 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Isaac Gym runs PhysX simulation, observations, rewards and PPO training on one GPU via a PyTorch tensor API; Shadow Hand cube reorientation drops from 30 h on a CPU cluster to under 1 h.

Why it matters

Tasks that previously needed a CPU cluster can be trained on a single GPU in minutes to hours, which shortens iteration for locomotion and dexterous-manipulation research. The authors also transfer policies trained this way to a real ANYmal on rough terrain and to a remote real TriFinger, indicating that the simulated contact physics supports sim-to-real work.

Problem

Robot RL usually simulates physics on CPUs and trains networks on GPUs, so data moves between devices at every step and simulation speed limits training. OpenAI's Shadow Hand cube work, for example, used 384 machines with 6144 CPU cores plus 8 V100 GPUs for 30 hours, and its Rubik's Cube follow-up used almost 30,000 CPU cores.

Proposed solution

Isaac Gym runs NVIDIA PhysX on the GPU with two additions: simulation results are no longer fetched to the CPU after every step, and a direct GPU API reads state and applies controls in GPU buffers. These buffers are wrapped as PyTorch tensors without copying, so thousands of parallel environments are stepped, observed, rewarded and trained (with the rl_games PPO implementation) on one GPU, with robots loaded from URDF or MJCF.

Key insights

Keeping physics state, observation and reward computation, and policy training on the GPU removes the CPU-GPU copies that limit conventional RL pipelines.
More parallel environments shorten training until the per-environment horizon gets too short: Humanoid gains nothing and learns worse gaits beyond 4096 environments at horizon 32, but learns with 8192 and 16384 environments when horizons are doubled.
Simulation throughput falls with model complexity: up to about 700K environment steps per second for Ant, 200K for Humanoid and 150K for Shadow Hand on an A100.
Asymmetric actor-critic, where the critic receives privileged simulator state and the policy only observations available on a real robot, is used for the Shadow Hand OpenAI and TriFinger setups to allow sim-to-real transfer.
A movable wrist helps reorientation: the Allegro Hand, with a fixed wrist and slightly longer fingers, reaches fewer consecutive successes than the Shadow Hand despite fewer degrees of freedom.

Main results

On a single A100: Ant passes a reward of 3000 in 20 s, Humanoid runs (reward 5000) in under 4 min, ANYmal follows velocity targets in under 2 min, and Franka cube stacking converges in under 25 min.
Shadow Hand with OpenAI observations and domain randomisation: more than 20 consecutive successes in under 1 h (feed-forward) and 37 in just under 6 h (LSTM), versus 30 h and about 17 h for OpenAI on 6144 CPU cores plus 8 V100 GPUs; the best LSTM seed reaches 37 in 2.5 h.
Shadow Hand Standard (no randomisation): 20 consecutive successes in under 35 min; raising environments from 256 to 16384 cuts training from 5×10⁴ s (about 14 h) to 3×10³ s (about 1 h).
AMP humanoid character animation: about 39 million samples in about 6 min with 4096 environments, versus about 30 h on 16 CPU cores in PyBullet, reported as a 300× improvement.
Sim-to-real: the ANYmal rough-terrain policy is trained on an RTX A6000 and transferred to the real robot in under 20 min; the real TriFinger reaches a 55% mean success rate, with transfer done remotely.
Conflicting speed-up figures: the v1 abstract states 1-2 orders of magnitude over CPU-simulator training, while the v1 introduction and Fig. 2 state 100×-1000×; the current arXiv abstract states 2-3 orders of magnitude.

Limitations

Shadow Hand experiments are simulation only; unlike OpenAI's work, no sim-to-real transfer is attempted for the hand, and the randomisations omit action delay and motor backlash.
Shadow Hand Standard uses observations that may not be available on a real robot, so the authors state it is not suitable for sim-to-real transfer.
Ingenuity is a simplified model: forces are applied directly to the two rotors instead of simulating aerodynamics.
Only limited tensor-API functionality is available with the FleX engine, scene setup runs on the CPU, and the included examples use PyTorch, with TensorFlow integration requiring further customisation.
Adding environments stops helping once horizons get short: Ant shows no gain from 8192 to 16384 environments, and Humanoid beyond 4096 environments at horizon 32 trains slower and learns sub-optimal gaits.

Field impact

1469 citations. The paper releases Isaac Gym as a download from NVIDIA (developer.nvidia.com/isaac-gym), including a basic PPO implementation and an RL task system, and hosts results and videos on its project site.

Abstract

Isaac Gym offers a high performance learning platform to train policies for wide variety of robotics tasks directly on GPU. Both physics simulation and the neural network policy training reside on GPU and communicate by directly passing data from physics buffers to PyTorch tensors without ever going through any CPU bottlenecks. This leads to blazing fast training times for complex robotics tasks on a single GPU with 2-3 orders of magnitude improvements compared to conventional RL training that uses a CPU based simulator and GPU for neural networks. We host the results and videos at https://sites.google.com/view/isaacgym-nvidia and isaac gym can be downloaded at https://developer.nvidia.com/isaac-gym.

Open Source Robotics