LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

Authors: Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, Lichao Sun · arXiv preprint 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

LIBERO-PRO perturbs LIBERO's objects, start positions, instructions and scenes. OpenVLA, pi0 and pi0.5 drop from 82–99% to near 0% under position and task changes.

Why it matters

Teams choosing or building a VLA policy often rely on LIBERO scores, and this paper shows those scores can hide complete failure when an object is moved or the instruction changes. It gives engineers a ready test suite to check whether a policy actually uses the scene and the instruction before trusting it on a robot.

Problem

LIBERO is the most widely reported benchmark for vision-language-action (VLA) models, but its evaluation tasks are the same as its training tasks, differing only by nearly invisible changes in the initial object states. A model that overfits can therefore score above 90% without understanding the scene or the instruction, which inflates results and makes comparisons between models unreliable.

Proposed solution

LIBERO-PRO is a plug-and-play extension of LIBERO that applies controlled perturbations along four dimensions: object attributes (new assets with changed appearance, size or colour), initial positions (alternative regions that respect physical constraints), instructions (three paraphrases per task plus redesigned tasks built from training objects and actions) and environments (swapping among LIBERO's five scene types). Perturbations can be combined at random, except task changes, which are kept separate to avoid meaningless combinations.

Key insights

Standard LIBERO test tasks differ from training tasks only by small changes in initial state, so memorized trajectories can score highly.
Models replay nearly the same trajectory when the target object is replaced by an unrelated one, moved or removed, and when the instruction is replaced by meaningless tokens such as 'xxx'.
Apparent robustness to paraphrased instructions is not evidence of language understanding, because the same models also ignore nonsense instructions.
A model that completes two atomic instructions separately failed when they were combined into one longer instruction.
Perturbations reveal differences that standard scores hide: pi0.5 keeps 0.38 success under position changes on libero-goal, where OpenVLA and pi0 score 0.

Main results

On the original LIBERO suites the three models average 0.82–0.99 success; under position perturbations OpenVLA and pi0 score 0.00 on all four suites.
Under task perturbations all three models score 0.00–0.01 on every suite (libero-goal, libero-spatial, libero-10, libero-object).
pi0.5 keeps some success under position changes: 0.38 (libero-goal), 0.20 (libero-spatial), 0.17 (libero-object) and 0.08 (libero-10).
Object-attribute and paraphrase perturbations barely change scores (e.g. OpenVLA on libero-goal: 0.98 original, 0.96 objects, 0.98 semantic), which the authors attribute to overfitting, not robustness.
Environment changes affect models unevenly: OpenVLA falls to 0.00 on libero-object, pi0 scores 0.27–0.60 and pi0.5 0.46–0.73 across suites.
OpenVLA and pi0 collapse to zero success once object displacement exceeds 0.2 units; pi0.5 keeps non-trivial success up to 0.4 units.

Limitations

The paper has no dedicated limitations section.
Only three models are evaluated (OpenVLA with four per-suite checkpoints, pi0 and pi0.5 with one checkpoint each), at 50 episodes per task.
All evaluation is in LIBERO simulation; no real-robot tests are reported.
Several findings (trajectory replay after object swaps, nonsense instructions, composed tasks) are shown through qualitative examples on selected tasks rather than aggregate statistics.
Displacement magnitudes are reported in simulator 'units' without a stated physical scale.

Field impact

253 citations. The authors release the benchmark code at github.com/Zxy-MLlab/LIBERO-PRO.

Abstract

LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluation settings are problematic, often leading to inflated performance estimates and preventing fair model comparison. To address these issues, we introduce LIBERO-PRO, an extended LIBERO benchmark that systematically evaluates model performance under reasonable perturbations across four dimensions: manipulated objects, initial states, task instructions, and environments. Experimental results reveal that, although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0% under our generalized setting. Crucially, this discrepancy exposes the models' reliance on rote memorization of action sequences and environment layouts from the training set, rather than genuine task understanding or environmental perception. For instance, models persist in executing grasping actions when the target object is replaced with irrelevant items, and their outputs remain unchanged even when given corrupted instructions or even messy tokens. These findings expose the severe flaws in current evaluation practices, and we call on the community to abandon misleading methodologies in favor of robust assessments of model generalization and comprehension. Our code is available at: https://github.com/Zxy-MLlab/LIBERO-PRO.

Open Source Robotics