What Are We Actually Benchmarking in Robot Manipulation?
Authors: Tianchong Jiang, Xiangshan Tan, Samuel Wheeler, Luzhe Sun, Tewodros W. Ayalew, Matthew Walter · CoRL 2026
Licences
- Open Access
- Paper: arXiv.org perpetual, non-exclusive license 1.0
Under arXiv's licence, everything below is written from the abstract only.
Why it matters
Benchmark scores measure success under one fixed setup but are routinely read as evidence of general manipulation capability. The audit finds that LIBERO and CALVIN fail several diagnostics: on LIBERO a 0.09B probe with no language encoder scores at or near reported state of the art, and on CALVIN randomizing block poses within the training range lowers every tested policy's performance.
Proposed solution
The authors name four failure modes (shortcut solvability, lack of statistical significance, creeping overfitting and data-source dependence), propose one diagnostic for each, apply them to five benchmarks, and release the diagnostics with reference implementations for authors and reviewers.
Abstract
A robotics benchmark score measures success under one fixed evaluation setup, yet is routinely treated as evidence of general manipulation capability. We identify four failure modes, each of which weakens or invalidates a benchmark's role as a valid proxy for that capability: shortcut solvability, lack of statistical significance, creeping overfitting, and data-source dependence. We propose one diagnostic per failure mode. We audit LIBERO, CALVIN, SimplerEnv, RoboCasa, and RoboTwin 2.0 under these diagnostics. LIBERO and CALVIN fail multiple diagnostics. RoboCasa and RoboTwin 2.0 fail fewer, despite appearing far less often in recent progress claims. On LIBERO, a 0.09B probe with no language encoder scores at or near reported SOTA, and most reported gains are not provably statistically significant. On CALVIN, randomizing block poses within the training range drops performance for every tested policy. We release the four diagnostics with reference implementations for authors and reviewers to apply before treating a benchmark score as evidence of progress. Code and artifacts are available at https://ripl.github.io/manipulation_benchmark_audit/.
Links
- arXiv 2606.04233: https://arxiv.org/abs/2606.04233
- PDF: https://arxiv.org/pdf/2606.04233
- Code: https://github.com/ripl/ManipulationBenchmarkAudit
- Project page: https://ripl.github.io/manipulation_benchmark_audit/