Geometric Action Model for Robot Policy Learning

Authors: Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong · arXiv 2026

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Splits a pretrained 3D geometry model and inserts a causal predictor so one backbone forecasts future depth and actions: 1.4B parameters, 6.9 ms per step, 83.1% under LIBERO-Plus camera shifts.

Why it matters

The policy has 1.4B parameters (about 983M trainable) and needs one feed-forward pass per step, giving about 145 Hz on a single GH200 GPU. Its largest gains are under camera perturbation, which matters when a deployed camera setup differs from the one used for data collection.

Problem

Vision-language-action models and video world-action models build on 2D image or 2D-derived latent priors, leaving depth, scale and occlusion implicit for the action decoder to work out. This limits generalization when the camera viewpoint or the robot's initial state changes. Prior geometry-aware VLAs use geometric foundation models only as static feature extractors.

Proposed solution

GAM splits a pretrained geometric foundation model (DA3-Giant) at layer 12: the shallow layers encode multi-view RGB, a 12-layer block-causal transformer inserted at the split predicts next-step latent geometry tokens and an action token from frozen-T5 language tokens, proprioception and the previous action, and the remaining deep layers decode both. A lightweight head regresses an 8-step action chunk and the original DPT head decodes future depth; training combines an ℓ1 action loss, a future-feature loss and a future-depth loss.

Key insights

Future prediction can sit inside a geometric model's own latent space: action and future-scene tokens share one autoregressive sequence and one backbone pass.
The split depth is decisive: Ls = 12 gives 99.6% / 70.1% (LIBERO / LIBERO-Plus Object, no depth loss), Ls = 19 stays competitive, and Ls = 0 or 27 and deeper collapse to near zero.
Pretraining matters most for robustness: without it and without future losses LIBERO-Plus Object success is 50.0% versus 89.7% for the full model, and the future-feature and depth losses alone recover part of that gap (73.4%).
A single-step history (H = 1) was more robust after post-training than 2 or 4 steps (89.7% versus 84.4% and 85.1% on LIBERO-Plus Object).
Passing the action token through the deep GFM blocks beats supervising it directly at the predictor output (89.7% versus 84.1% on LIBERO-Plus Object).

Main results

LIBERO 97.6% and LIBERO-Plus 85.5% (a 12.1-point drop), the highest LIBERO-Plus score in the comparison; π0.5 scores 96.9% / 84.6% and Cosmos-Policy 98.5% / 82.4%.
LIBERO-Plus camera perturbation: 83.1% versus 73.4% for the next best method (Cosmos-Policy), a 9.7-point gain.
Latency on one GH200: 6.9 ms per step with CUDA Graphs and 17.5 ms with Torch Compile only, versus 29.2 ms for π0.5 and 382.4 ms for Cosmos-Policy; OpenVLA-OFT is listed at 77.8 ms in the main-text table and 70.1 ms in the appendix table.
RoboCasa-Kitchen (24 tasks, 3 views): 69.4% average versus 67.1% for Cosmos Policy.
Real robot, four tasks with 10 nominal and 10 camera-shifted trials each: 65% in-distribution average versus 55% for π0.5 and 35% for Spatial Forcing.
With the external camera moved 85 cm and rotated 45°, Figure 1 reports 52.5% for GAM versus 30% and 22.5%; the per-task bars in Figure 4 (80, 60, 20, 40) average 50%.

Limitations

The authors state that language reasoning and commonsense ability are bounded by the frozen text encoder; adding a large language model or an external reasoning module is left as a next step.
Real-world evaluation is small: four tasks, 20 trials each, trained from 169–284 demonstrations per task; Place pot and pan reaches only 30% in-distribution and 20% under camera shift.
In the real world, future-depth targets are pseudo-labels from the backbone itself because ground-truth depth is unavailable.
On LIBERO-Plus, GAM trails π0.5 under robot-initial-state (70.0% vs 76.6%), language (84.8% vs 86.5%) and layout (79.1% vs 86.0%) perturbations.
Pretraining used 64 NVIDIA GH200 GPUs for about 96 hours.

Field impact

On Hugging Face daily papers it drew 110 upvotes, in the top 5% of robotics papers there over the last six months. The paper gives a project page (cvlab-kaist.github.io/Geometric-Action-Model); its text does not state a code or checkpoint release.

Abstract

Generalist robot policies must follow user instructions while reasoning about how objects, cameras, and robot actions interact in the 3D physical world. Recent vision-language-action models (VLAs) and video world-action models (WAMs) inherit strong semantic or temporal priors from large-scale foundation models, but they still operate primarily on 2D image frames or 2D-derived latent spaces, leaving implicit the 3D geometry required for contact-rich manipulation. We propose the Geometric Action Model (GAM), a language-conditioned manipulation policy that directly repurposes a pretrained geometric foundation model (GFM) as a shared substrate for perception, temporal prediction, and action decoding. GAM splits the GFM at an intermediate layer: the shallow layers serve as an observation encoder, and a causal future predictor inserted at the split layer forecasts future latent tokens conditioned on language, proprioception, and action history. The predicted future tokens are then routed through the remaining GFM blocks for feature propagation and decoding, allowing a single backbone to produce both future geometry and actions. This design equips the GFM with language-conditioned temporal world modeling through minimal architectural modification while preserving its rich geometric priors. Across a broad suite of simulation and real-robot manipulation benchmarks, GAM is more accurate, more robust, faster, and lighter than current foundation-model-scale baselines.

Open Source Robotics