Advancing Humanoid Locomotion: Mastering Challenging Terrains with Denoising World Model Learning

Authors: Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, Jianyu Chen · RSS 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

DWL trains a humanoid walking policy in Isaac Gym that learns to rebuild the full robot state from noisy proprioception, then runs unchanged on real XBot robots over stairs, slopes and snow.

Why it matters

The same policy, using only joint readings, IMU data, a clock signal and velocity commands with no camera or LiDAR, transferred zero-shot to two humanoid sizes and handled stairs, slopes, irregular blocks, snow, added loads and pushes. For legged-robot engineers it is a concrete recipe for getting state estimation (base velocity, foot contact, a rough terrain profile) from the same network that produces actions. It also gives a direct comparison showing that actively controlled 2-DoF ankles mattered on the harder terrains.

Problem

Humanoids have a high centre of gravity, heavy legs, torso and arms, and an unstable leg-swing phase, and the authors note that earlier real-world RL controllers for humanoids were limited to relatively simple terrain. A policy trained in simulation must also cope with noisy sensors, imperfect simulated dynamics and quantities such as linear velocity and contact force that the real robot cannot measure.

Proposed solution

Denoising World Model Learning (DWL) treats the sim-to-real gap as noise on the true state: in training it masks unobservable quantities and applies domain randomization, and a GRU encoder compresses the observation history into a 24-dimensional latent from which a decoder must rebuild the full state, including base velocity, friction, foot contact and a terrain height scan. The actor acts on that latent and is trained with PPO while an asymmetric critic sees the full state, with denoising, policy and value losses optimised together. The policy actively controls both ankle degrees of freedom through a closed kinematic chain, and a quintic-polynomial foot trajectory gives a dense reward for soft foot contact.

Key insights

Modelling the sim-to-real gap as environmental, dynamics, sensory and masking noise, and training an encoder-decoder to remove it, gives the policy a built-in state estimator.
From proprioception alone, the latent state holds enough information to recover a rough terrain height profile, foot contacts and base velocity, and the gait changes when the robot moves from a slope to stairs.
Active control of both ankle DoF through a closed kinematic chain was decisive on irregular terrain and stairs; Isaac Gym cannot model the closed chain, so two virtual motors are simulated and their targets remapped to the real motors.
Rewarding foot-velocity tracking along a planned quintic-polynomial swing trajectory, rather than relying only on sparse contact forces, gives a reward signal at every step and encourages gentle touchdowns.

Main results

On XBot-S, DWL succeeded 100% on slope, stair-up, stair-down and irregular terrain; PPO with ankle control scored 80%, 20%, 60% and 20%, and DWL with passive ankles 80%, 20%, 100% and 40%.
The robot kept walking with up to 15 kg added progressively to a carried bag (over a third of its weight), walked with up to 20 kg extra load, and pushed a cart loaded with 60 kg.
In a MuJoCo sim-to-sim test, forward-velocity estimation MSE over 60 s was 0.046 and Euler-yaw estimation MSE was 0.074; on the real robot, IMU drift was reduced by about 87%.
One policy, running at 100 Hz with a 500 Hz PD loop, transferred zero-shot to XBot-S (1.2 m, 38 kg) and XBot-L (1.65 m, 57 kg), including outdoor walks on cement, brick, soil and snow.
The DWL actor has about 320,192 trainable parameters and was trained with 12,288 parallel simulated environments.

Limitations

The policy uses no visual input; the authors leave adding visual information for navigation on challenging terrain to future work.
The authors state that precise terrain shape prediction from proprioception remains challenging; only a rough profile is recovered.
True states cannot be measured on the real robot, so velocity and yaw estimation accuracy was checked in a MuJoCo sim-to-sim transfer rather than on hardware.
Benchmark comparisons were run only on XBot-S with the arms held still, and the paper does not state how many trials each success rate is based on.
Leg control only: the arm motors were kept stationary throughout the study.

Field impact

Finalist for the RSS 2024 Outstanding Paper Award and cited 152 times. The paper publishes its full network architecture, reward terms, domain-randomization ranges and hyperparameters in the appendix. It does not state a code, model or hardware-design release.

Abstract

Humanoid robots, with their human-like skeletal structure, are especially suited for tasks in human-centric environments. However, this structure is accompanied by additional challenges in locomotion controller design, especially in complex real-world environments. As a result, existing humanoid robots are limited to relatively simple terrains, either with model-based control or model-free reinforcement learning. In this work, we introduce Denoising World Model Learning (DWL), an end-to-end reinforcement learning framework for humanoid locomotion control, which demonstrates the world's first humanoid robot to master real-world challenging terrains such as snowy and inclined land in the wild, up and down stairs, and extremely uneven terrains. All scenarios run the same learned neural network with zero-shot sim-to-real transfer, indicating the superior robustness and generalization capability of the proposed method.

Open Source Robotics