Inspect Robots

Define a robotics benchmark once, run any policy on any robot

License

MIT

Overview

Inspect Robots is an open evaluation framework for physical AI. You define a robotics benchmark once, then run any policy against any compatible embodiment: an LLM agent or a VLA, on a real arm, a humanoid, or a simulator. Every run produces auditable logs — grader scores, the full LLM transcript and the complete config — and there is first-class Rerun visualisation so a run can be watched rather than only tabulated. The project describes itself as Inspect AI for robotics, and the comparison is apt: the value is in the harness and the record-keeping rather than in any particular task.

The structure that matters is the separation between benchmark, policy and rig. A benchmark definition does not know what will run it; a policy does not know what it is running on; a rig plugin supplies the embodiment. That is what makes 'run any policy against any embodiment' more than a slogan — installing inspect-robots-yam, for example, brings both a molmoact2 policy and a yam_arms rig, and the same benchmark then runs unchanged. There is first-class support for SO-100 and SO-101 arms that reuses LeRobot calibration, which is the difference between a framework a hobbyist can use and one that assumes a lab.

Engineering standards are unusually visible for a project this young: CI, mypy strict, and badges claiming 100% test and public-docstring coverage, with Python 3.10 through 3.13 and a documentation site at docs.inspectrobots.org. It installs with uv or pip, and the rerun extra pulls in the live run viewer. The authors are equally visible about maturity — the README carries an alpha badge and asks you to pin a version, because the API may change between releases.

It is also the harness behind a published result: Robocurve used it to run GPT-6 Astra and Claude Fable 5.1 on the same YAM arms and released all 120 trials, which is the best available demonstration that the logging is genuinely auditable.

Why it matters

Claims about models controlling robots are usually unfalsifiable: a demo video, no harness, no trial count, no way to rerun it. Shared evaluation tooling is what turned language-model claims into something arguable, and robotics has not had the equivalent. This is a credible attempt at one, and crucially it is cheap to adopt — a reader with an SO-101 on a desk can install it and run the same benchmark a lab ran, which is the property that decides whether an evaluation framework spreads.

Problem it solves

There is no shared way to ask whether a policy is actually any good at a physical task, so every claim arrives on its own terms and none of them can be compared. Inspect Robots supplies the harness, the grading and the audit trail, and keeps them independent of the robot so the same question can be asked of different hardware.

Key takeaways

• Benchmark, policy and embodiment are separate concerns, so the same benchmark runs against an LLM agent or a VLA, on a real arm, a humanoid or a simulator
• Every run emits auditable logs: grader scores, the full LLM transcript and the complete config
• First-class Rerun visualisation means a run can be watched, not just scored
• First-class SO-100 and SO-101 support reuses LeRobot calibration, so low-cost arms are a supported target rather than an afterthought
• Rig plugins bundle policy and embodiment together — inspect-robots-yam supplies a molmoact2 policy and a yam_arms rig
• CI, mypy strict and badges claiming 100% test and docstring coverage, across Python 3.10-3.13
• It is the harness behind Robocurve's published GPT-6 Astra evaluation, where all 120 trials were released with traces

Who it is for

• Anyone evaluating an LLM agent or VLA on a real robot and wanting the result to be checkable
• SO-100 and SO-101 owners who want to benchmark a policy on hardware they already have
• Research groups standardising how they report manipulation results
• People reproducing or contesting a published robot evaluation

Limitations

• Explicitly alpha: the README says the API may change between releases and asks you to pin a version before depending on it
• The set of supported rigs is small — YAM arms and SO-100/101 are what is documented, so any other embodiment means writing a plugin
• Coverage and typing badges are self-reported by the project's own CI rather than independently audited
• Grading in the published use of it was done by a human on a rubric, so the framework standardises the record-keeping rather than removing the human from scoring
• Built and maintained by the same group that publishes evaluations with it, which is efficient but means the tooling and the results are not independent of each other
• Young: the repository dates from late June 2026