Evaluating Real-World Robot Manipulation Policies in Simulation

Authors: Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, Ted Xiao · CoRL 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

SIMPLER: open-source simulated versions of the Google Robot and WidowX/BridgeData V2 setups whose policy rankings track real results (MMRV 0.031, r 0.976 on Pick Coke Can).

Why it matters

Teams iterating on imitation-learning policies get a cheap, repeatable signal that tracks real performance, and the paper shows validation action MSE is a poor proxy for it (Table I). The environments install with pip, use the standard Gym interface, and render 3.5k simulation steps per second on one RTX 4090, which the authors give as a 7× speed-up over real evaluation.

Problem

Generalist manipulation policies must be tested across many tasks and scenes, and real-robot evaluation at that breadth is expensive, slow and hard to reproduce between labs. Building full digital twins for simulation is also costly, since it needs hand-made geometry, materials, lighting and physical properties.

Proposed solution

The authors build simulations that are only realistic enough for simulated and real success rates to rank policies the same way. They close the control gap by identifying the simulated controller's stiffness and damping (simulated annealing on end-effector tracking error against existing demonstration data), and the visual gap by 'green-screening' a real background behind simulated foreground objects and matching object and robot-arm textures to real images. They also propose the Mean Maximum Rank Violation (MMRV) metric, which weights each ranking error by the real-world performance gap between the mis-ranked policies.

Key insights

An exact digital twin is not needed: the simulation only has to preserve the relative ranking of policies.
Pearson r alone can mislead, since it rewards a linear fit and swings with small noise when policies perform similarly; MMRV weights each rank violation by the real performance gap.
Visual fixes help only when applied together: green-screened background plus texture-matched drawer and robot gave the lowest drawer-task MMRV (0.050), while partial combinations did no better than the 0.087 baseline.
Controller system identification matters: perturbed stiffness and damping raised Pick Coke Can MMRV from 0.031 to 0.070–0.100.
Simulation can predict real sensitivities not tested before, such as Octo-Base's sensitivity to robot arm texture, which a real experiment with gift-wrapped arms then confirmed.

Main results

Google Robot with Visual Matching: MMRV 0.031 / r 0.976 on Pick Coke Can, 0.111 / 0.855 on Move Near and 0.000 / 0.969 on Open Drawer and Place Apple. For Open/Close Drawer, Fig. 6 reports 0.055 / 0.915 while the Drawer column of Table I reports 0.027 / 0.942; the paper does not explain the difference.
Averaged over Google Robot tasks (Table I): Visual Matching MMRV 0.056 and r 0.924, Variant Aggregation 0.143 and 0.778, validation action MSE 0.375 and 0.308.
WidowX with BridgeData V2 tasks: averaged MMRV 0.014 and r 0.890; all policies were ranked correctly on all but one task.
Distribution shifts on two RT-1 checkpoints: MMRV 0.000 / r 0.831 without data augmentation and 0.016 / 0.970 with it; new solid table colours cut real success by 4% and simulated by 2%, patterned textures by 25% real and 24% simulated.
Reproduced in Isaac Sim with Variant Aggregation on Pick Coke Can and Move Near: MMRV 0.058 / r 0.919, versus 0.082 / 0.923 in SAPIEN.
Octo-Base scored 0% in simulation with the untuned arm texture and 29.3% with a tuned one; changing object mass, gripper friction or cabinet joint friction moved success rates by up to 15% while correlations stayed high.

Limitations

Only rigid-object manipulation is covered; deformable objects are left for future work with soft-body simulation.
Green-screening is limited to fixed cameras and does not reproduce object shadows and other visual details.
Creating environments still needs manual work to curate assets and assemble scenes; articulated objects such as the cabinet take the most effort.
Physical properties are simplified (uniform densities looked up with GPT-4 or web search, generic friction values), and absolute success rates are not expected to match the real world, only rankings.
Real evaluation of one policy on the drawer tasks was stopped after 2 trials for risk of damaging the robot, so that MMRV is an upper estimate.

Field impact

Cited 426 times. The authors release all SIMPLER environments, policy inference code for RT-1, RT-1-X and Octo, and a guide for building new environments at simpler-env.github.io, plus command-line texture-matching scripts (github.com/Jiayuan-Gu/GeTex).

Abstract

The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, which are likely to worsen as policies broaden the spectrum of tasks they can perform. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to craft full-fidelity digital twins of real-world environments. We then employ these approaches to create SIMPLER, a collection of simulated environments for manipulation policy evaluation on common real robot setups. Through paired sim-and-real evaluations of manipulation policies, we demonstrate strong correlation between policy performance in SIMPLER environments and in the real world. Additionally, we find that SIMPLER evaluations accurately reflect real-world policy behavior modes such as sensitivity to various distribution shifts. We open-source all SIMPLER environments along with our workflow for creating new environments at https://simpler-env.github.io to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks.

Open Source Robotics