Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Authors: Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, Liyiming Ke · CoRL 2026

CoRL 2026

Licences

  • Open Access
  • Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence.

TL;DR

RL Token (RLT) bootstraps online RL on top of a pretrained VLA via a single compact readout token plus a small actor-critic head — fine-tuning real-robot policies in minutes-to-hours with up to 3× speed gains.

Why it matters

Most VLA papers focus on pretraining or imitation learning. The hard problem in deploying VLAs is the last-mile gap between "kind of works" and "production-ready precision and speed." RLT shows this gap can be closed in minutes-to-hours of real-robot practice rather than weeks of additional data collection, and does so without per-architecture custom RL machinery (the way π-RL had to). This makes RL post-training a credible deployment step rather than a research curiosity.

Problem

Pretrained VLAs (RT-2, π₀, OpenVLA, π₀.5) can execute manipulation skills out-of-the-box but lack the precision and speed real-world tasks demand. Existing online RL fine-tuning of VLAs is either slow (high sample complexity), brittle (catastrophic forgetting), or architecture-specific (e.g. π-RL only works with flow-matching policies).

Proposed solution

Adapt the base VLA to expose a compact "RL token" — a readout that summarises task-relevant pretrained knowledge. Train a small actor-critic head on top of this token while keeping the VLA frozen (or KL-anchored). The token provides a stable, low-dimensional interface for policy gradient methods, decoupling RL fine-tuning from the underlying VLA architecture.

Key insights

  • A single "RL token" prepended to the VLA's input acts as a stable, compact interface for online RL — sidestepping the flow-matching log-probability problem that π-RL had to work around.
  • The base VLA stays frozen (or KL-anchored). Only a small actor-critic head trains, which keeps sample complexity low and prevents catastrophic forgetting of pretrained skills.
  • Online RL is finished in minutes to a few hours of real-robot practice — short enough that engineers can iterate physically without simulating.
  • On precision-bottlenecked tasks (screw alignment, Ethernet insertion) RLT can surpass human teleoperation speed — the policy learns a smoother trajectory than the human can produce manually.
  • The approach generalises across four heterogeneous real-robot tasks without per-task hyperparameter tuning, suggesting the token-based interface is task-agnostic.

Main results

  • Up to 3× speed-up on the hardest portion of four real-robot manipulation tasks within minutes to a few hours of online RL practice.
  • On at least one task, RLT surpasses human teleoperation speed — the fine-tuned policy executes the precision insertion faster than a human operator.
  • Sample efficiency is high enough that engineers can iterate on a real robot rather than relying on simulation transfer.
  • The same token-based recipe works across four heterogeneous tasks (screw installation, zip tie fastening, charger insertion, Ethernet insertion) without per-task retuning.

Limitations

  • Evaluated on a single robot platform across four manipulation tasks — generalisation to mobile manipulation / locomotion / multi-arm settings is left to future work.
  • Requires a strong pretrained VLA as the base; the method does not address how to train that base model in the first place (which is where most of the cost still lives).
  • Online RL with real hardware still demands engineering effort (reset mechanisms, safety constraints, reward shaping) — the paper does not claim push-button autonomy.

Field impact

Establishes a clean, model-agnostic recipe for the "post-training" step that has been missing from the VLA stack. Sits alongside HIL-SERL (2024), RL-100 (2025), and π-RL (2025) in the emerging RL Post-Training & Deployment era — extending the trend of efficient online fine-tuning while removing architecture-specific constraints. Likely to be the template subsequent VLA papers (π₀.5+, GR00T+) cite when describing their deployment pipeline.

Abstract

Vision-language-action (VLA) models can learn to perform diverse manipulation skills "out of the box," but achieving the precision and speed that real-world tasks demand requires further fine-tuning — for example, via reinforcement learning (RL). We introduce a lightweight method that enables sample-efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. We (1) adapt the VLA to expose an "RL token," a compact readout representation that preserves task-relevant pretrained knowledge while serving as an efficient interface for online RL, and (2) train a small actor-critic head on this RL token to refine the actions, while anchoring the learned policy to the VLA. Online RL with the RL token (RLT) makes it possible to fine-tune even large VLAs with RL quickly and efficiently. Across four real-robot tasks (screw installation, zip tie fastening, charger insertion, and Ethernet insertion), RLT improves the speed on the hardest part of the task by up to 3x and raises success rates significantly within minutes to a few hours of practice. It can even surpass the speed of human teleoperation on some of the tasks.

Links

  • arXiv 2604.23073: https://arxiv.org/abs/2604.23073
  • PDF: https://www.pi.website/download/rlt.pdf
  • Project page: https://www.pi.website/research/rlt

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.