Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

uni-robot-agent

Open harness that lets open-weight language models control robots through shared tools

License

MIT

Overview

uni-robot-agent is an MIT-licensed harness for robot-use agents: language models that control a robot by calling tools for perception and motion rather than outputting joint commands. Released by Tsinghua University and UC Berkeley with the paper 'Democratizing Robot-Use Agents: A Unified Harness with Open Models' (November 2026), it gives every environment the same tool interface, runs agents through Codex or Claude Code runtimes or its own Python SDK, and records each episode for replay in a local dashboard. The public release covers three simulation integrations; physical-robot support for ARX One and Franka is announced but not yet released.

uni-robot-agent connects a vision-capable language model to a robot through one set of shared tools. The agent reasons over camera images, calibration and the robot's description, then calls perception tools (images to object positions) and motion tools (targets to robot actions); adapters translate those calls into each environment's native API and return execution receipts as feedback. An executor overlaps the agent's reasoning with tool execution while keeping actions on the same arm in order. In Direct mode the agent drives the robot through these tools; Hybrid mode hands low-level control to a compatible action model such as π₀.₅.

The release (v0.1.1 on PyPI, Python 3.11+, Linux GPU host for the simulators) supports three integrations: RoboDojo, a dual-arm manipulation benchmark; RoboUse, with selected tasks from LIBERO, RoboCasa, Meta-World, robosuite and ManiSkill plus MuJoCo Menagerie and Genesis adapters; and RoboLab, single-arm DROID tasks in Isaac Lab. By default it runs in the paper's realistic setting, without simulator depth or task-verifier feedback to the agent.

In the paper, the open-weight Qwen3.8-Flash-Next reaches a RoboDojo Score of 31.39 zero-shot through this harness, against 1.90 for the same model under the earlier RoboProbe harness and 28.97 for the published GPT-6 Astra–RoboProbe result. On 19 physical-robot tasks it scores 53.50, against 47.75 for π₀.₅ fine-tuned with 10 hours of data per task and 43.50 for zero-shot Dream-1; the authors note their agents used D405 sensor depth while both baselines used RGB only.

Why it matters

Robot-use agents built on frontier models such as GPT-6 Astra have shown that a general model with good tools can manipulate objects zero-shot, but closed weights mean researchers cannot train those models to get better at it. uni-robot-agent shows that open-weight models already get competitive results when the harness gives them reliable tools and informative feedback, and it records every episode in one format across environments, which is what training open robot-use agents needs. The authors' analysis across model generations, families and sizes ties success to four capabilities, visual understanding, spatial grounding, geometric coding and embodied reasoning, giving training work concrete targets. It is published as an installable package with documentation rather than only as paper code (GitHub).

Problem it solves

Research on robot-use agents has depended on proprietary models reached through paid APIs, harnesses tied to one simulator, and setups that hand the agent privileged simulator information. uni-robot-agent provides one open harness with shared tools across environments, runs open-weight or hosted models through any Responses-compatible endpoint, keeps the agent to the information a real robot would have, and saves interaction records that can be reused for training.

Key takeaways

  • One tool interface for perception, motion, skills and episode control across simulators, with adapters for each environment's native API
  • Runs agents through Codex or Claude Code runtimes or a Python SDK, with any Responses-compatible model endpoint
  • Direct mode drives the robot through tools; Hybrid mode adds an action model such as π₀.₅
  • Three simulation integrations: RoboDojo, RoboUse (LIBERO, RoboCasa, Meta-World, robosuite, ManiSkill and others) and RoboLab on Isaac Lab
  • Paper result: open-weight Qwen3.8-Flash-Next scores 31.39 zero-shot on RoboDojo, above the published GPT-6 Astra–RoboProbe result of 28.97

Who it is for

  • Researchers training or evaluating open-weight models as robot-use agents
  • Robot-learning groups comparing language-model agents with VLA policies on the same tasks
  • Developers who want to drive simulated robots from an LLM through a documented Python SDK
  • Benchmark builders adding new environments behind a shared tool interface

Limitations

  • Simulation only in the public release: the ARX One and Franka integrations used for the paper's 19 real-world tasks are listed as coming soon
  • The simulators need a supported Linux GPU host, and the base package installs no simulators or action-model checkpoints
  • Results depend on the model you connect; the agent needs a vision-capable model behind a Responses-compatible endpoint, which for hosted models means per-call costs
  • Released in November 2026 at version 0.1.1, so its interfaces may still change
  • The real-world comparison is not like-for-like: the agents used sensor depth while the π₀.₅ and Dream-1 baselines used RGB only

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.