Don't Blind Your VLA: Aligning Visual Representations for OOD Generalization

Authors: Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov · arXiv 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Fine-tuning OpenVLA on robot actions collapses its visual representations; aligning layer-16 features to a frozen C-RADIOv3 teacher raises OOD success in Simpler, e.g. Carrot 0.49 → 0.61.

Why it matters

The method adds only precomputed teacher features and one loss term to standard fine-tuning, so a team adapting a pretrained VLA to its own robot with little data can try it without changing the model. The study also shows that freezing the vision encoder during fine-tuning is not a safe shortcut: that baseline failed almost completely.

Problem

VLAs are expected to inherit vision-language knowledge from their pretrained VLMs, but task-specific supervised fine-tuning on limited robot data can erode it. The authors find that OpenVLA's attention becomes diffuse, its object embeddings collapse into overlapping clusters, and it scores below its base PrismaticVLM in most domains of a new diagnostic suite.

Proposed solution

Visual Representation Alignment adds a regularizer to LoRA fine-tuning: vision-token hidden states from a middle backbone layer (layer 16) of OpenVLA pass through a frozen MLP projector and are pulled toward patch features of a frozen C-RADIOv3 teacher with a cosine-similarity loss weighted by λ = 0.2. The paper also introduces VL-Think, eight board-selection tasks in Simpler with a WidowX-250S arm that isolate vision-language understanding from control difficulty.

Key insights

Naive action fine-tuning causes attention sink and representation collapse in OpenVLA compared with its base VLM.
Forgetting is domain-specific: OpenVLA-7B falls below PrismaticVLM on arrows (0.26 vs 0.47), public-information signs (0.24 vs 0.62), traffic signs and weather icons, but keeps colour (0.69 vs 0.69).
Aligning middle backbone layers to the teacher (Backbone2Enc) beats aligning the VLA's own vision encoder (Enc2Enc), which places the degradation in the vision-language fusion layers.
A frozen projector works best; a trainable one lets the model reduce the alignment loss inside the projector instead of changing its own representations.
Freezing the visual encoder during fine-tuning gave near-zero success (0.01–0.05) in every environment.

Main results

Semantic OOD, SFT → Align (128 seeds): Carrot 0.49 → 0.61, Instruct 0.74 → 0.83, MultiCarrot 0.28 → 0.35, MultiPlate 0.43 → 0.49.
Vision OOD: VisionImg 0.81 → 0.86, Tex05 0.55 → 0.67, Whole03 0.71 → 0.79 (0.80 in the appendix tables).
Execution OOD: Position 0.43 → 0.58 and EEPose 0.34 → 0.38, but PosChangeTo 0.23 → 0.20.
ImageNet-100 linear probe: OpenVLA SFT 77.48%, pretrained OpenVLA 79.88%, OpenVLA Align 82.13%, C-RADIOv3 teacher 87.31%.
VL-Think: alignment raises Color from 0.69 to 0.82 and Shape from 0.40 to 0.48, leaving most other domains roughly unchanged.
Teacher ablation (mean Semantic / Vision / Execution): C-RADIOv3 0.61 / 0.72 / 0.39, DINOv2 0.57 / 0.69 / 0.37, Theia 0.56 / 0.67 / 0.36, SigLIP 0.54 / 0.65 / 0.35.

Limitations

The authors state that, due to compute constraints, the study covers fine-tuning only, not full-scale pretraining.
The authors note alignment only partly reverses domain forgetting on VL-Think, which they attribute to the modest size and diversity of the fine-tuning set and the limited expressivity of LoRA updates.
All experiments are in simulation (Simpler-based environments with a WidowX-250S arm) and use one model, OpenVLA-7B; there is no real-robot test.
Execution-axis gains are small, PosChangeTo drops slightly, and several ablation comparisons are not statistically significant (p > 0.05).

Field impact

On Hugging Face daily papers it drew 107 upvotes, in the top 5% of robotics papers there over the last six months. The paper states that its code is publicly available via blind-vla-paper.github.io and introduces the VL-Think diagnostic suite.

Abstract

The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. In this work, we conduct a systematic study of representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, we probe VLA’s hidden representations and analyze attention maps, further, we design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. We further evaluate a range of strategies for aligning visual representations and introduce a simple yet effective method that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios. Taken together, our analysis clarifies the trade-off between action fine-tuning and the degradation of VL representations and highlights practical approaches to recover inherited VL capabilities. Code is publicly available: blind-vla-paper.github.io

Open Source Robotics