Learning Vision-Based Bipedal Locomotion for Challenging Terrain

Authors: Helei Duan, Bikram Pandit, Mohitvishnu S. Gadde, Bart van Marum, Jeremy Dao, Chanho Kim, Alan Fern · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A sim-trained RL policy plus a depth-to-heightmap predictor lets the Cassie biped cross blocks, stairs and a 0.5 m step-up without global odometry or real-world fine-tuning.

Why it matters

The whole system runs onboard and does not estimate global pose, which removes a common failure source for legged robots that strike the ground hard and often. The authors report it as the first sim-to-real demonstration of vision-based bipedal locomotion over challenging terrain, and they document which training choices were needed for the transfer to work.

Problem

RL controllers for bipeds have mostly been blind, relying on proprioception alone, so they cannot anticipate terrain and tend to use aggressive gaits with high swing trajectories. Model-based perceptive pipelines split the problem into whole-body control, footstep planning, odometry and terrain mapping, and maps built in a global frame drift with pose-estimation error caused by frequent contacts and aggressive motion.

Proposed solution

A locomotion policy is trained with PPO in MuJoCo on a 1.5 m by 1 m heightmap at 5 cm resolution, expressed relative to the robot's floating base; it combines a pretrained LSTM blind policy with a vision-based modulator that also adjusts gait frequency and per-leg phase shifts. A separate heightmap predictor (an LSTM stage followed by a U-Net refinement stage) is trained on 30,000 simulated episodes to map a history of egocentric depth images and robot states to that heightmap. Domain randomization of dynamics, torque efficiency, delays, heightmap noise and depth-image noise carries both networks to the real robot.

Key insights

Expressing the heightmap in the robot's local frame lets the policy react to terrain without global mapping or odometry; on hardware the policy still placed swing feet correctly over terrain the camera could no longer see, which the authors read as the policy keeping an internal odometry.
The terrain sampling mix (mostly blocks, ridges and stairs) shaped a robust, non-aggressive gait more effectively than adding heuristic reward terms: stairs regulate step length and ridges regulate step height.
Without a foot-collision penalty the policy reached similar success in simulation but used collisions as feedback, for example sliding a foot up a vertical face; the resulting frequent stumbles prevented sim-to-real transfer.
Randomizing torque efficiency was particularly important for the 0.5 m step-up, where the knee motor saturates.
A learned gait clock and a critic given privileged inputs (global feet positions, heightmaps around each foot, foot-tip rangefinders) both raised success on the hard terrain settings in simulation.

Main results

On hardware, Cassie crossed single high blocks, stairs, random blocks and a 0.5 m step-up (about 60% of leg length) at speeds up to 1 m/s, and kept speed and heading on a treadmill while a person fed blocks onto it.
Heightmap reconstruction error in simulation (MAE): LSTM 2.806 cm, Transformer 4.221 cm, MLP 4.932 cm, LSTM without robot states 4.448 cm.
In closed-loop simulation all heightmap predictors gave similar success rates, but the LSTM predictor produced the fewest foot-collision episodes and the fewest collision-caused terminations.
On the robot the depth stream ran at 90 FPS, heightmap inference alone ran at 200 Hz on the Jetson Orin Nano, and end-to-end delay between predictor and policy was measured at up to 20 ms.
In simulation ablations (1000 episodes per terrain mode), policies without the learned clock or without the privileged critic had markedly lower success on hard terrain, and the blind policy performed worst.

Limitations

Because only one forward-facing camera is used, velocity commands on ridge, stair and block terrain exclude backward and sideways walking.
The egocentric camera cannot see the ground underneath the robot, so terrain under the feet has to be handled from what the policy has already seen.
Hardware results are shown qualitatively (lab wooden blocks, stairs, a treadmill and the submission video); the paper gives no real-world success rates.
Training and evaluation terrains are limited to five procedurally generated types: flat, hills, blocks, ridges and stairs.
The foot-collision penalty acts only as a soft constraint, which the policy may violate to avoid falling.

Field impact

Finalist for the IEEE ICRA 2024 Best Paper Award in Cognitive Robotics, with 98 citations. The paper does not link released code, data or hardware designs.

Abstract

Reinforcement learning (RL) for bipedal locomotion has recently demonstrated robust gaits over moderate terrains using only proprioceptive sensing. However, such blind controllers will fail in environments where robots must anticipate and adapt to local terrain, which requires visual perception. In this paper, we propose a fully-learned system that allows bipedal robots to react to local terrain while maintaining commanded travel speed and direction. Our approach first trains a controller in simulation using a heightmap expressed in the robot’s local frame. Next, data is collected in simulation to train a heightmap predictor, whose input is the history of depth images and robot states. We demonstrate that with appropriate domain randomization, this approach allows for successful sim-to-real transfer with no explicit pose estimation and no fine-tuning using real-world data. To the best of our knowledge, this is the first example of sim-to-real learning for vision-based bipedal locomotion over challenging terrains.

Open Source Robotics