LingBot-Map

Streaming 3D reconstruction from one camera: pose and depth for every frame, live

License

Apache 2.0

Code and model weights: Apache-2.0 (repository LICENSE.txt and Hugging Face model card). The backbone is initialised from DINOv2 (Apache-2.0). The training mix includes academic datasets with their own terms, some non-commercial (for example ScanNet and the Waymo Open Dataset); the authors release the weights under Apache-2.0 regardless. Checked 2026-10-23.

Overview

LingBot-Map is a feed-forward model from Robbyant, Ant Group's embodied-AI lab, that reconstructs a scene in 3D from a single moving camera as the video arrives. For each new frame it predicts the camera's pose and a depth map, using only the frames it has already seen, so the trajectory and a dense point cloud build up live, much as a visual SLAM system would, but without hand-built tracking, feature matching or bundle adjustment. The code and weights are Apache-2.0 in the lingbot-map repository, with checkpoints on Hugging Face, and the work is described in a technical report and an ECCV 2026 paper. The paper reports about 20 FPS at 518 × 378 pixels on an NVIDIA H800 and about 12 FPS on an RTX 4090, over sequences of more than 10,000 frames. It comes from the same lab as LingBot-VLA and LingBot-VA.

LingBot-Map follows the design of recent 3D foundation models such as VGGT, which predict cameras and geometry for a set of images in one pass, and makes it causal so it can run on a stream. A DINOv2 image backbone feeds a transformer that alternates attention within each frame with Geometric Context Attention across frames. That attention keeps three kinds of memory: a few anchor frames from the start, which fix the coordinate frame and scale; a sliding window of recent frames for dense local geometry; and a compact per-frame trajectory memory that summarises everything older to limit drift. Because the state stays small, the cost per frame stays nearly constant as the sequence grows. The streaming implementation uses a paged key-value cache through FlashInfer, with a plain PyTorch fallback.

The repository ships an interactive demo that streams a folder of images or video frames into a browser viewer (viser), an offline renderer for long sequences, keyframe and windowed modes for runs beyond about 3,000 frames, sky masking for outdoor scenes, and the evaluation scripts for the benchmarks in the paper, including KITTI, Oxford Spires, TUM and ETH3D. Three checkpoints are published: the balanced model used in the paper, a long-sequence model, and the stage-1 checkpoint, which loads into VGGT for offline, bidirectional reconstruction.

The model was trained in two stages on 31 dataset entries, first on short multi-view clips and then on long video trajectories, including ground-robot, aerial and driving data such as TartanGround, MidAir, KITTI-360 and Waymo. The paper reports about two weeks on 64 GPUs for the first stage.

Why it matters

A robot that knows where its camera is and what surrounds it can navigate, map and plan. Classical visual SLAM gets there with carefully engineered tracking and optimisation that tends to break on texture-poor scenes and fast motion, and learned offline reconstruction models are accurate but need the whole video in advance. LingBot-Map is one of the first open models to do learned reconstruction causally, frame by frame, at a usable rate over long trajectories, with weights under a licence that permits commercial use. That makes it a practical drop-in to try for camera-only mapping on a robot or drone, or for building 3D maps from robot video logs.

Problem it solves

Getting camera poses and dense 3D geometry from an ordinary camera in real time, over long trajectories, without a hand-built SLAM pipeline and without waiting for the whole video. Earlier streaming learned models drifted or slowed down as sequences grew; LingBot-Map keeps a bounded memory so it stays fast and stable over thousands of frames.

Key takeaways

• Predicts a camera pose and a depth map for each frame of a video as it arrives, using only past frames
• About 20 FPS at 518 × 378 on an NVIDIA H800 and about 12 FPS on an RTX 4090, per the paper, over 10,000+ frame sequences
• Bounded memory: anchor frames, a sliding window and a compact trajectory memory keep per-frame cost nearly constant
• Code and all three checkpoints are Apache-2.0, built on a DINOv2 backbone
• Comes with a browser viewer demo, long-sequence and windowed modes, and benchmark scripts for KITTI, Oxford Spires and others
• From Robbyant, the lab behind LingBot-VLA and LingBot-VA

Who it is for

• Robotics engineers who want camera-only mapping or odometry for a wheeled robot, legged robot or drone
• Researchers comparing learned reconstruction with classical visual SLAM
• Teams building 3D maps from recorded robot or vehicle video
• Anyone needing camera trajectories and depth from video for downstream training or simulation

Limitations

• Needs a capable NVIDIA GPU: the paper's 20 FPS is on an H800, about 12 FPS on an RTX 4090, and each checkpoint is about 4.6 GB
• No explicit loop closure or test-time optimisation, so drift can still build up on very long or revisited routes
• Beyond the trained range, long runs need keyframe or windowed mode, which resets state between windows and adds drift
• Monocular camera only for now: no IMU, lidar or stereo input, and no handling of lens distortion
• It is a reconstruction model, not a full robot stack: there is no ROS integration and no relocalisation against an earlier map, though reconstructions can be exported as GLB
• Weights are Apache-2.0, but the training mix includes academic datasets with their own non-commercial terms, such as ScanNet and Waymo