Navigation World Models

Authors: Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, Yann LeCun · CVPR 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A 1B-parameter diffusion transformer predicts future egocentric frames from past frames and navigation actions, then plans or ranks paths by comparing the imagined final frame with a goal image.

Why it matters

Planning by simulation lets a navigation system add constraints at plan time and sample more candidates for harder goals, which fixed policies cannot do. On RECON, standalone NWM planning lowered ATE to 1.13 from 1.87 (GNM) and 1.93 (NoMaD), and using NWM to rank NoMaD samples improved NoMaD on all four in-domain datasets.

Problem

Current visual navigation policies such as NoMaD and GNM fix their behavior after training: new constraints like 'no left turns' cannot be added, and they cannot spend more computation on harder problems. A world model that simulates where actions lead could address both, but a standard diffusion transformer's attention cost grows quadratically with the number of context frames.

Proposed solution

NWM learns to predict a future latent frame from up to four past frames plus a translation, a yaw rotation and a time shift of up to ±16 s, trained on robot navigation video (SCAND, TartanDrive, RECON, HuRoN) and optionally unlabeled Ego4D video. Its Conditional Diffusion Transformer (CDiT) restricts self-attention to the frame being denoised and attends to past frames through cross-attention, so cost is linear in context length. For planning, the Cross-Entropy Method searches action sequences that minimize LPIPS between the imagined last frame and the goal image, with constraints as penalties; alternatively NWM ranks trajectories sampled from NoMaD.

Key insights

Restricting self-attention to the target frame and reading past frames through cross-attention makes the cost linear in context length while improving prediction over a standard DiT.
Conditioning on both actions and a time shift works best: time-only conditioning fails (LPIPS 0.760) while action plus time gives 0.295 on RECON at 4 s.
Sampling four goals per state during training and using four context frames both improve prediction; with one context frame the model often loses track.
Unlabeled Ego4D video improves prediction in an unseen environment (Go Stanford) but lowers accuracy in most known environments, SCAND being the exception.
Constraints such as 'turn first, then go forward' are imposed simply by zeroing parts of the action sequence during planning.

Main results

Goal-conditioned navigation on RECON (Table 2, 2-s trajectories): standalone NWM planning ATE 1.13 and RPE 0.35, versus 1.87/0.73 for GNM and 1.93/0.52 for NoMaD; ranking 32 NoMaD samples with NWM gives 1.78/0.48. Appendix Table 5 lists slightly different RECON values for NoMaD (1.95/0.53) and for ranking (1.88/0.51 at ×16, 1.79/0.49 at ×32).
Video quality, 16-s videos at 4 FPS: FVD 200.969 for NWM versus 762.734 for DIAMOND on RECON, and 276.9 / 494.2 / 401.7 versus 882.0 / 2289.7 / 1945.1 on HuRoN, TartanDrive and SCAND.
Best 4-s prediction on RECON (4 goals, 4 context frames, action plus time): LPIPS 0.295, DreamSim 0.091, PSNR 15.343.
Planning under three action constraints met every constraint, with final-position differences of +0.36, −0.03 and +0.08 and yaw differences of +0.61, +0.20 and +0.22 relative to unconstrained planning.
Adding unlabeled Ego4D video improves the unseen Go Stanford set (LPIPS 0.658 to 0.652, DreamSim 0.478 to 0.464) but worsens RECON (LPIPS 0.295 to 0.368).
Compute: the introduction says CDiT needs 4× fewer FLOPs than a standard DiT; Section 4.2 says CDiT performs better up to 1B parameters 'while consuming less than ×2 FLOPs' and that CDiT-L is ×4 faster than DiT-XL. The XL model was trained on 8 machines with 8 H100 GPUs each.

Limitations

On out-of-distribution input the model slowly loses context and generates frames resembling the training data (mode collapse).
It struggles to simulate temporal dynamics such as pedestrian motion, though sometimes it succeeds.
Actions are 3-DoF navigation on a flat surface; 6-DoF navigation and arm control are left for future work.
Standalone planning was tested on 2-s, 8-step trajectories with a single CEM iteration and a straight-line action parameterization.
All evaluation is offline on recorded datasets with no closed-loop runs on a physical robot; on HuRoN, standalone planning ATE (4.12) is worse than GNM (3.71) and NoMaD (3.73) even though the Table 5 caption says NWM improves all metrics.

Field impact

The paper received a CVPR 2025 Best Paper Honorable Mention and has 400 citations. Its official project page links code (github.com/facebookresearch/nwm) and model weights on Hugging Face (facebook/nwm).

Abstract

Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.

Open Source Robotics