H2-Mapping: Real-Time Dense Mapping Using Hierarchical Hybrid Representation

Authors: Chenxing Jiang, Hanwen Zhang, Peize Liu, Zehuan Yu, Hui Cheng, Boyu Zhou, Shaojie Shen · RA-L 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Coarse SDF stored in an octree plus a hash-encoded residual lets NeRF-style RGB-D mapping run onboard Jetson Orin boards, on a handheld rig and a quadrotor.

Why it matters

A robot with only a Jetson-class computer can build a textured dense map while it moves, instead of reconstructing offline afterwards. Because the method takes poses from any external tracker, it can sit behind an existing visual-inertial odometry stack, as the authors show with a modified VINS-Mono and a RealSense L515.

Problem

NeRF-based mappers such as NICE-SLAM and Vox-Fusion give compact maps and can render new viewpoints, but their implicit features take a long time to optimise, so they cannot run in real time even on an AGX Orin. Explicit maps (occupancy grids, TSDF, surfels, meshes) are fast but need a lot of memory for fine detail and cannot render realistic novel views.

Proposed solution

Coarse geometry is stored explicitly as SDF values at the vertices of a sparse voxel octree (10 cm leaf voxels), initialised by projecting each vertex into the depth image; a multiresolution hash encoding with small MLPs learns only the residual SDF and the colour. Voxels are expanded when points sit on a voxel edge so surfaces are not lost, keyframes are chosen to cover as many not-yet-optimised voxels as possible, and optimisation stops early when the loss stops improving.

Key insights

Projecting voxel vertices into a single depth image already gives a usable coarse SDF, so geometry is close to right from the first iterations
Letting the hash encoding learn only the residual SDF is easier than learning the full SDF and raises both accuracy and rendering quality for the same number of iterations
Allocating an extra voxel when all back-projected points lie on a voxel edge prevents holes where an SDF value near zero flips sign during optimisation
Selecting keyframes to cover the most unoptimised voxels, then resetting once all are covered, improves seldom-seen regions such as ceilings without adding training samples

Main results

Replica (8 scenes, ground-truth poses): mean Depth L1 0.42 cm vs 0.71 cm for Vox-Fusion and 1.16 cm for NICE-SLAM; completion ratio (<5 cm) 99.25% vs 98.55% and 97.09%
Replica rendering: mean PSNR 34.49 dB on training views and 30.21 dB on distant novel views, vs 31.48 / 27.34 dB for Vox-Fusion and 30.55 / 28.08 dB for NICE-SLAM
Mean frame processing time on Replica Room0: 0.05 s on an RTX 4090, 0.36 s on AGX Orin and 0.66 s on Orin NX, vs 0.34 / 1.68 / 3.52 s for Vox-Fusion and 2.69 / 13.01 / 19.03 s for NICE-SLAM
Ablation (Table III): removing the keyframe strategy drops PSNR from 34.49 to 33.65 dB and worsens accuracy from 1.12 to 1.28 cm; removing voxel expansion drops completion ratio from 99.25% to 98.52%
Real-world runs fully onboard: an apartment of about 127 m² mesh surface mapped with a handheld AGX Orin rig, and part of a flight arena of about 58 m² mapped with a quadrotor carrying an Orin NX

Limitations

The authors state the method cannot handle dynamic objects
The authors state it cannot handle long-term pose drift
The authors state further speed-up is still required
It is a mapping module only: poses must come from a separate tracker, and the benchmark comparisons use ground-truth poses for all methods
ScanNet results are qualitative only, because ground-truth meshes are not available for that dataset

Field impact

Received the RA-L 2024 Best Paper Award and has 58 citations. The paper states that its code will be released at github.com/SYSU-STAR/H2-Mapping.

Abstract

Constructing a high-quality dense map in real-time is essential for robotics, AR/VR, and digital twins applications. As Neural Radiance Field (NeRF) greatly improves the mapping performance, in this paper, we propose a NeRF-based mapping method that enables higher-quality reconstruction and real-time capability even on edge computers. Specifically, we propose a novel hierarchical hybrid representation that leverages implicit multiresolution hash encoding aided by explicit octree SDF priors, describing the scene at different levels of detail. This representation allows for fast scene geometry initialization and makes scene geometry easier to learn. Besides, we present a coverage-maximizing keyframe selection strategy to address the forgetting issue and enhance mapping quality, particularly in marginal areas. To the best of our knowledge, our method is the first to achieve high-quality NeRF-based mapping on edge computers of handheld devices and quadrotors in real-time. Experiments demonstrate that our method outperforms existing NeRF-based mapping methods in geometry accuracy, texture realism, and time consumption. The code will be released at: https://github.com/SYSU-STAR/H2-Mapping

Open Source Robotics