GeoVLA: Empowering 3D Representations in Vision-Language-Action Models

Authors: Lin Sun, Bin Xie, Yingfei Liu, Hao Shi, Tiancai Wang, Jiale Cao · IROS 2026

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

GeoVLA feeds a depth-derived point cloud, encoded around the end-effector, into a diffusion action head next to VLM features, and holds up better under height, size and viewpoint changes.

Why it matters

For engineers fitting VLAs to real arms, the paper shows a way to use an RGB-D camera's depth without retraining the VLM on 3D data. The gains over 2D baselines are largest exactly where those baselines fail in the real tests: changed target heights, object sizes and camera viewpoints.

Problem

Most vision-language-action models act from 2D RGB images only, so they miss the depth and geometric cues needed for precise spatial reasoning. Earlier ways of adding 3D either put 3D positional encodings into the VLM, which breaks its alignment with the visual encoder and needs large 3D instruction-tuning datasets, or freeze the action expert and inject point-cloud features afterwards, which limits adaptation to the new modality.

Proposed solution

GeoVLA keeps a pretrained VLM (Prismatic-7B, initialised from OpenVLA weights) for image and language, and in parallel turns the depth map into a point cloud in the end-effector frame, encoded by a Point Embedding Network that uses large-kernel convolutions, rotary positional encoding and the end-effector token as an anchor. A diffusion-transformer action head, the 3D-enhanced Action Expert, fuses both feature sets through modality-specific experts with static routing; it is trained end to end while one modality is randomly dropped in each iteration.

Key insights

Encoding the point cloud in a separate branch, rather than inside the VLM, keeps the VLM's pretrained vision-language alignment intact.
Expressing the point cloud in the end-effector frame and using the end-effector token as the anchor gave 97.7% on LIBERO, versus 96.3% for max pooling and 95.9% for mean pooling.
Dynamic MoE routing biased the action head toward the pretrained VLM branch; static routing with random modality dropout gave 97.7%, versus 97.3% for dynamic routing and 96.0% without MoE.
Rotary positional encoding (RoPE) in the point encoder raised the LIBERO average from 95.4% (1D learnable embeddings) to 97.7%.
The 3D branch matters most for out-of-distribution geometry: with the camera rotated 45°, GeoVLA kept 70% success while CogACT fell to 0%.

Main results

LIBERO: 97.7% average over five suites, versus 95.3% for OpenVLA-OFT and 93.2% for CogACT; 96.6% on LIBERO-Long and 97.7% on LIBERO-90, using only one main camera view plus its point cloud.
ManiSkill2 (five pick-and-place tasks, 20 episodes each): 77% average versus 69% for CogACT and 66% for Dita; 45% on PickClutterYCB versus 36% for Dita.
Real world, 8 tasks on a WidowX-250s arm, 10 trials each: 86.3% average versus 76.3% for CogACT, 57.5% for π0 and 20.0% for OpenVLA (95.0% on basic tasks, 77.5% on 3D-aware tasks).
Basket one layer higher than in training: 60% versus 20% for both π0 and CogACT; largest unseen doll size: 80% versus 50% for CogACT and 0% for π0.
Camera viewpoint shifted by 15°, 30° and 45°: 90%, 80% and 70%, versus 60%, 40% and 0% for CogACT.
Point-encoder ablation on LIBERO: Point Embedding Network 97.7% versus 95.8% for a 3-layer MLP (DP3-style) and 95.2% for PointNet.

Limitations

The paper has no limitations section of its own; the points below come from its reported results and setup.
With the sponge mat removed in Pick Carrot (carrot sits lower), GeoVLA reached 50% while π0 reached 60%, although the text says GeoVLA achieves more stable grasps than the other methods.
Real-world tests use one arm (WidowX-250s), one third-person RealSense camera and 10 trials per setting.
Lighting changes still hurt: Stack Block fell from 80% to 50% under a light change, and the three-task average fell from 93.3% to 73.3% (appendix Table 9).
On ManiSkill2 the model also receives proprioceptive state, gripper state and the target marker position, and on PickCube (90%) it stays below CogACT (95%).

Field impact

Finalist for the IROS 2026 Best Paper Award on Cognitive Robotics, sponsored by KROS, with 84 citations. The official project page links a code repository at github.com/linsun449/geovla.code.

Abstract

Vision-Language-Action (VLA) models have emerged as a promising approach for enabling robots to follow language instructions and predict corresponding actions. However, current VLA models mainly rely on 2D visual inputs, neglecting the rich geometric information in the 3D physical world, which limits their spatial awareness and adaptability. In this paper, we present GeoVLA, a novel VLA framework that effectively integrates 3D information to advance robotic manipulation. It uses a vision-language model (VLM) to process images and language instructions, extracting fused vision-language embeddings. In parallel, it converts depth maps into point clouds and employs a customized point encoder, called Point Embedding Network, to generate 3D geometric embeddings independently. These produced embeddings are then concatenated and processed by our proposed spatial-aware action expert, called 3D-enhanced Action Expert, which combines information from different sensor modalities to produce precise action sequences. Through extensive experiments in both simulation and real-world environments, GeoVLA demonstrates superior performance and robustness. It achieves state-of-the-art results in the LIBERO and ManiSkill2 simulation benchmarks, and shows remarkable robustness in real-world tasks requiring height adaptability, scale awareness and viewpoint invariance. The project is available at https://linsun449.github.io/GeoVLA.

Open Source Robotics