RoboDojo

Manipulation benchmark that runs in Isaac Sim and on real arms

License

MIT

Overview

RoboDojo is a benchmark for generalist manipulation policies that covers simulation and real hardware in one place: 42 Isaac Sim tasks across five capability dimensions, and 18 real-world tasks on Piper X, Piper and ARX X5 arms. The piece that makes it usable is XPolicyLab, which unifies more than 40 policies behind a single interface so the same policy definition runs in simulation and on a real robot. Layouts are seed-controlled for reproducibility and a single summarize command aggregates runs into a leaderboard table.

The catalogue already carries LIBERO, RLBench, Meta-World, CALVIN and SimplerEnv, and every one of them stops at the simulator. RoboDojo's distinguishing claim is that the real half is not an afterthought: 18 tasks on three real arm platforms sit alongside the 42 simulated ones, under the same policy interface. Whether a policy's simulated score predicts its real score is the question the field most needs answered and least often measures, and a benchmark built to ask it directly is worth having.

Practically, the repository separates the simulator backbone and managers, the environment configuration for simulator, scene, robot and camera, the task logic and YAML task configs, and XPolicyLab as the policy server and integration layer. Assets — robots, objects, materials and layouts — are downloaded rather than vendored. The documentation site is treated as the canonical reference and covers installation, quick evaluation, integrating a new policy into XPolicyLab, the sim and real task lists, configuration and troubleshooting, which is more attention to the getting-started path than most benchmarks manage.

Why it matters

Simulated manipulation benchmarks have become easy to saturate and hard to trust: a number on LIBERO says little about whether a policy will work on a bench. RoboDojo puts both halves under one interface so the comparison can actually be made, and it does it with 40-plus policies already integrated, which is the difference between a benchmark you can use this week and one you would have to port your work into first.

Problem it solves

Manipulation results are reported in simulation and deployed on hardware, and nobody can say how much the first predicts the second because the two are measured with different tooling. RoboDojo puts them behind one policy interface so the same thing can be run in both places.

Key takeaways

• 42 Isaac Sim tasks across five capability dimensions, plus 18 real-world tasks on Piper X, Piper and ARX X5 arms
• XPolicyLab unifies 40+ policies behind one interface, so the same definition runs in simulation and on real hardware
• Seed-controlled layouts and a one-command summarize step aggregate runs into a leaderboard table
• Sim and real are designed as two halves of one benchmark rather than a simulator with a real demo attached
• Assets are downloaded rather than vendored, keeping the repository itself light
• The documentation site is the canonical reference and covers integrating new policies, not just running existing ones

Who it is for

• Groups reporting generalist manipulation results who want a real-robot number alongside the simulated one
• Anyone comparing VLA policies and tired of each paper using its own harness
• Owners of Piper, Piper X or ARX X5 arms with a policy to evaluate
• Researchers integrating a new policy who want it runnable across many tasks at once

Limitations

• The real half needs specific hardware — Piper X, Piper or ARX X5 — so most readers can only run the simulated 42
• Isaac Sim is a heavy dependency with its own GPU requirements, and the docs carry a troubleshooting page for GPU memory for a reason
• Assets must be downloaded separately, so the benchmark is not self-contained
• The paper behind it (arXiv 2607.04434) is a preprint with no venue, so the benchmark's design has not been through review
• A leaderboard is only as good as its participation, and at this age that is unproven
• XPolicyLab integrating 40+ policies is a maintenance burden that will show if the project slows down