Robot Parkour Learning

Authors: Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn, Hang Zhao · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Two-stage RL (penetrable obstacles first, real collisions second) trains five parkour skills; DAgger distills them into one depth-image policy that runs onboard low-cost Unitree A1 and Go1 robots.

Why it matters

The policy runs using only onboard computation, onboard depth sensing and onboard power on low-cost quadrupeds, with no reference motion data, and the same training pipeline drives two robots (A1 and Go1). In the authors' comparisons, the soft-constraint pre-training is what made climbing and leaping learnable at all.

Problem

Parkour needs a robot to perceive obstacles and pick among diverse agile skills at speed, but prior learning methods give either diverse but blind skills or vision-based but specialized ones, using many-term reward functions or animal motion-capture data that has no egocentric vision. For low-cost small robots the obstacles are large relative to body size, onboard depth perception lags during fast motion (0.2 s of delay at 1 m/s is 0.2 m of error), and parkour drives the motors to their limits.

Proposed solution

Each skill (climb, leap, crawl, tilt, run) is trained in IsaacGym with PPO from privileged obstacle information and one reward for forward speed near 1 m/s, mechanical energy and staying alive. Stage one makes obstacles penetrable and penalizes penetration volume and depth under an automatic curriculum (soft dynamics constraints, inspired by direct collocation); stage two fine-tunes with real collisions. DAgger then distills the five specialists into one GRU policy that takes proprioception and a 48×64 depth image, deployed with matched depth pre-processing and a 25 Nm torque clip.

Key insights

Letting the robot pass through obstacles early, with a penetration penalty and an automatic curriculum, solves the exploration problem: without it, and with the RND exploration baseline, no climbing or leaping was learned.
One reward made of forward progress, squared mechanical power and an alive bonus produced all five skills without per-skill reward tuning or motion data.
The policy needs memory: when climbing, the rear legs must act on obstacle dimensions seen in earlier depth images, so the GRU policy succeeds where an MLP policy fails.
Distilling in action space with DAgger merges specialists with different network weights into one vision policy, which RMA-style latent-space distillation cannot do.
Depth images were matched between simulation (clipping, Gaussian noise, random artifacts) and the real camera (clipping, hole-filling, spatial and temporal filters); a re-trying behaviour on failed climbs emerged without being programmed.

Main results

Real A1 indoor tests, 10 trials per setting: 80% success climbing 0.40 m (1.53× robot height), 80% leaping 0.60 m gaps (1.5× robot length), 90% crawling under 0.2 m (0.76× robot height), and squeezing through 0.28 m slits by tilting; best of all compared methods, including the A1's built-in MPC controller.
Simulation success of the distilled parkour policy (100 trials × 3 seeds, test obstacles harder than training): climb 86%, leap 80%, crawl 100%, tilt 73%, run 100%.
Baselines in simulation: Blind 0% climb, leap and tilt and 13% crawl; MLP 0% climb and 1% leap; No Distill 0% climb, leap and tilt; RMA, run only on tilt, 74%.
Specialist oracles trained with soft-dynamics pre-training reached about 95% average success over climb, leap, crawl and tilt; without it, 0% on climb and leap.
Outdoors the robot leapt a 0.6 m gap between two stone stools, climbed successive 0.42 m steps and crawled beneath a camping cart.
Training cost: each specialist 12 h with soft dynamics plus 6 h with hard dynamics on one machine with an Nvidia 3090; distillation on four machines, each with one 3090.

Limitations

The simulation environments are constructed by hand, so new skills can only be learned when new obstacle environments are added to the simulation (stated by the authors).
The policy uses depth only; training from RGB with semantic information is left to future work (stated by the authors).
Onboard depth refreshes at only 10 Hz because of limited onboard computation, while the policy runs at 50 Hz on the latest available depth embedding.
Quantitative real-world tests were run on the A1 only, at 10 trials per setting; no real-world success rate is stated for the 0.28 m tilt task.

Field impact

Finalist for the CoRL 2023 Best Systems Paper Award and cited 356 times. The authors present it as an open-source system for training and deploying agile locomotion policies; the project page links the code repository.

Abstract

Parkour is a grand challenge for legged locomotion that requires robots to overcome various obstacles rapidly in complex environments. Existing methods can generate either diverse but blind locomotion skills or vision-based but specialized skills by using reference animal data or complex rewards. However, autonomous parkour requires robots to learn generalizable skills that are both vision-based and diverse to perceive and react to various scenarios. In this work, we propose a system for learning a single end-to-end vision-based parkour policy of diverse parkour skills using a simple reward without any reference motion data. We develop a reinforcement learning method inspired by direct collocation to generate parkour skills, including climbing over high obstacles, leaping over large gaps, crawling beneath low barriers, squeezing through thin slits, and running. We distill these skills into a single vision-based parkour policy and transfer it to a quadrupedal robot using its egocentric depth camera. We demonstrate that our system can empower two different low-cost robots to autonomously select and execute appropriate parkour skills to traverse challenging real-world environments.

Open Source Robotics