GRaD-Nav++: Vision-Language Model Enabled Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics

Authors: Qianzhong Chen, Naixiang Gao, Suning Huang, JunEn Low, Timothy Chen, Jiankai Sun, Mac Schwager · IEEE Robotics and Automation Letters (RA-L) 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A compact VLA for quadrotors, trained by differentiable RL in a 3D Gaussian Splatting simulator with a mixture-of-experts head, runs fully onboard and follows language instructions.

Why it matters

It shows that a compact language-conditioned flight policy can run entirely on a small onboard computer at 25 Hz, with no external infrastructure. Training takes about 7 hours on one RTX 4090, and the simulation-trained policy flies on real hardware without additional tuning.

Problem

Language-guided drones either use an LLM to choose from predefined skills through API calls, leaving a gap to low-level control, or rely on VLA models with billions of parameters that cannot run on a drone's onboard computer and need ground-station links. Earlier drone VLAs were limited to racing or were evaluated only in simulation.

Proposed solution

GRaD-Nav++ encodes the instruction and the drone's RGB view with a frozen CLIP model, run asynchronously every 10 control steps, and combines this with min-pooled depth, a beta-VAE context estimate and the drone state. A two-expert mixture-of-experts policy maps these inputs to body-rate and thrust commands. It is trained with differentiable RL in a photorealistic 3D Gaussian Splatting scene coupled to a differentiable quadrotor model, so policy gradients flow through the dynamics.

Key insights

Differentiable RL through 3DGS rendering and quadrotor dynamics learned the multi-task language policy, while PPO with the same VLM failed within 1x10^7 steps.
The VLM grounds instructions in the visual scene rather than just encoding text: replacing it with one-hot task codes gave 0/24 real-world success on trained tasks.
A small mixture-of-experts head (two experts, top-2 routing) avoided catastrophic forgetting when training alternated between environments, where same-size and larger MLPs did not.
Running CLIP asynchronously and using no separate CNN for RGB-D keeps onboard inference fast; depth is reduced to a 32-value min-pooled vector for collision avoidance.
When the instruction changes mid-flight, the CLIP text-image similarity drops and then recovers as the drone re-grounds, and the policy completes the new task.

Main results

Multi-task generalisation in simulation: 83% (20/24) success on trained tasks and 75% (9/12) on unseen tasks; PPO scored 0% and the no-VLM ablation 4% (1/24) and 0%.
On real hardware: 67% (16/24) success on trained tasks and 50% (6/12) on unseen tasks.
Multi-environment adaptation: 81% average success across simulated environments; the abstract gives 67% across real-world settings, while the paper's real-world table counts 18 of 24 trials (75%).
In the harder mid-gate environment the MoE policy kept 70% success in simulation and 67% on hardware, while single-expert (25% / 8%) and large-network (33% / 25%) baselines dropped sharply.
Training took about 7 hours (one environment, 800 epochs) or about 14 hours (two environments) on one RTX 4090; VLM inference added about 5% wall-clock time over GRaD-Nav.

Limitations

Generalises only to new combinations of sub-tasks seen in training; it fails when both sub-tasks are new (e.g. 'FLY OUT the LEFT door then LAND on the table'), so it is not an open-vocabulary solution (authors).
Model capacity and the task set are small: 12 two-stage tasks (8 trained) and two 3DGS environments (authors cite these as the cause of the limit above).
On hardware, Stage 2 (approaching the named object) loses more success than Stage 1, which the authors attribute to detection, fine control and error accumulation.
Each training task needs a reference trajectory from key waypoints placed on the 3DGS point cloud and connected by A* planning.
Real-world evaluation is small, with 3 trials per task.

Field impact

Winner of the RA-L 2026 RA-L Best Paper Award, with 26 citations so far. The licensed arXiv version does not link code, data or a project page.

Abstract

Autonomous drones capable of interpreting and executing high-level language instructions in unstructured environments remain a long-standing goal. Yet existing approaches are constrained by their dependence on hand-crafted skills, extensive parameter tuning, or computationally intensive models unsuitable for onboard use. We introduce GRaD-Nav++, a lightweight Vision-Language-Action (VLA) framework that runs fully onboard and follows natural-language commands in real time. Our policy is trained in a photorealistic 3D Gaussian Splatting (3DGS) simulator via Differentiable Reinforcement Learning (DiffRL), enabling efficient learning of low-level control from visual and linguistic inputs. At its core is a Mixture-of-Experts (MoE) action head, which adaptively routes computation to improve generalization while mitigating forgetting. In multi-task generalization experiments, GRaD-Nav++ achieves a success rate of 83% on trained tasks and 75% on unseen tasks in simulation. When deployed on real hardware, it attains 67% success on trained tasks and 50% on unseen ones. In multi-environment adaptation experiments, GRaD-Nav++ achieves an average success rate of 81% across diverse simulated environments and 67% across varied real-world settings. These results establish a new benchmark for fully onboard Vision-Language-Action (VLA) flight and demonstrate that compact, efficient models can enable reliable, language-guided navigation without relying on external infrastructure.

Open Source Robotics