NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields

Authors: Arunkumar Byravan, Jan Humplik, Leonard Hasenclever, Arthur Brussee, Francesco Nori, Tuomas Haarnoja, Ben Moran, Steven Bohez, Fereshteh Sadeghi, Bojan Vujatovic, Nicolas Heess · ICRA 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A NeRF trained on a 5-6 minute phone video supplies the visuals and collision mesh for a MuJoCo simulation; RGB-only humanoid policies trained there transfer zero-shot to a real robot.

Why it matters

The pipeline needs only a commodity phone video to recreate a visually realistic scene with usable contact geometry, and it trains whole-body bipedal control from RGB images alone, without localisation or a separate low-level controller. On the navigation task, success and time-to-goal on the real robot closely matched simulation.

Problem

Sim2real training needs simulations that match how a robot sees and touches a real scene, but modelling the geometry and appearance of unstructured scenes is slow and usually relies on specialised 3D scanning rigs that work poorly outdoors. RGB cameras have a large sim2real gap, so earlier visual navigation transfer work leaned on depth sensors or LiDAR, localisation, and existing low-level controllers.

Proposed solution

A short phone video is split into about 1000 keyframes, localised with COLMAP, and used to train a NeRF; the NeRF renders the robot's camera view and its occupancy is turned into a collision mesh with marching cubes, scaled and aligned to the simulator, with the floor replaced by a flat plane. MuJoCo simulates the robot and known dynamic objects (a ball), whose renderings are overlaid on the NeRF image, and DMPO trains recurrent vision-and-proprioception policies with an asymmetric critic, domain randomisation and image augmentation.

Key insights

A single NeRF supplies both the photoreal rendering and the static contact geometry for a physics simulation, so one phone video is enough to build a training scene.
Overlaying simulator renderings of known dynamic objects (robot body, ball) on the NeRF image is enough for the tested tasks, at the cost of not handling occlusions.
Accurate sensor and actuator models plus limited domain randomisation (pushes, latency, added mass, IMU offset) were enough for the gait to transfer.
Image augmentation cannot replace camera calibration: the navigation policy was highly sensitive to the camera gain setting.
Policies trained end-to-end still kept a stable gait when the scene changed or the camera was blocked, suggesting locomotion and scene understanding were partly separated.

Main results

Navigation, 60x80 input: 87±5% success on the real robot (47/54 episodes) vs 86±2% in simulation; median time 11.2±0.4 s real vs 10.8±0.1 s sim.
Navigation, 30x40 input: 69±6% success on the real robot (37/54 episodes) vs 73±3% in simulation; median time 10.3±0.4 s real vs 10.3±0.3 s sim.
Median localisation error of the policy's pose belief: 0.24±0.01 m and 0.27±0.04 m on the real robot vs 0.17±0.01 m and 0.19±0.004 m in simulation (30x40 and 60x80).
Ball pushing, 30x40 input: 78±7% real success with the ball starting in the centre and 43±14% with it near a wall, vs 99% and 100% in simulation.
With camera gain changed from the default 50 (4/5 successes) to 90 or 128, the navigation policy failed in 0/5 trials, walking in a small circle.
The blur augmentation for low-resolution NeRF rendering is reported as about 31.5 vs about 35.4 average PSNR on held-out images; the hash-grid NeRF renders one frame in 6 ms on a V100 GPU.

Limitations

Dynamic objects are always rendered in front of the static NeRF scene, so occlusions are not handled; more complex dynamic scenes would need composite NeRF rendering.
Manual steps remain: picking floor points in Blender, aligning and scaling the mesh, and replacing the poorly textured floor with a flat plane.
Going from a video to a trained policy takes 1-2 days (COLMAP about 3-4 hours, policy training about 24 hours for navigation and twice that for ball pushing).
The sim2real gap is larger for the contact-rich ball pushing task, especially when the ball starts near a wall.
On-board compute (2 CPU cores, no GPU, 25 ms per step) limited policies to 30x40 and 60x80 images, and the head-tilt range was capped because the ceiling is poorly captured by the NeRF.

Field impact

Finalist for the ICRA 2023 Outstanding Robot Learning Paper award, with 80 citations. The paper releases a project website with videos; no code release is stated.

Abstract

We present a system for applying sim2real approaches to "in the wild" scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dynamics from the static scene geometry (estimated from the NeRF volume density) and the dynamic objects' geometry and physical properties (assumed known). We demonstrate that we can use this simulation to learn vision-based whole body navigation and ball pushing policies for a 20 degrees of freedom humanoid robot with an actuated head-mounted RGB camera, and we successfully transfer these policies to a real robot. Project video is available at https://sites.google.com/view/nerf2real/home.

Open Source Robotics