wuji-mjlab
Train and deploy in-hand cube and pen-spinning policies on the Wuji Hand
License
Apache-2.0
Overview
wuji-mjlab is Wuji Technology's open training and deployment code for dexterous in-hand manipulation on its own robot hands. Policies are trained with PPO in mjlab, the GPU-parallel robot-learning framework built on MuJoCo, and run on the physical hand in a closed loop with a camera. Two tasks ship. Cube Reorient turns a cube held palm-up to any target orientation and holds it there, on both the Wuji Hand and the Wuji Hand 2. PenSpin makes the Wuji Hand 2 spin a 250 mm pen by tracking reference motions, controlling 20 finger joints at 50 Hz. The September 2026 release bundles pretrained checkpoints as PyTorch and ONNX, 40 pen-spinning motion clips with their training recipes, and printable mounting jigs, cubes and pens as STEP and 3MF files. The code is under the Apache 2.0 licence.
Most in-hand manipulation results are a paper and a video. This one is a working pipeline you can run end to end. Training, export and deployment are all in the repository, the policies are downloadable, and the physical rig is specified down to the camera lens, the tag size and the mounting jig. The deploy side is deliberately light. The reorient policy runs on numpy and onnxruntime alone, with no PyTorch or mjlab at run time. It is split into three processes joined by ZeroMQ: a camera observer that estimates the cube's pose from fiducial tags, the policy loop that drives the hand, and an optional MuJoCo mirror for watching what the policy thinks is happening.
The first release in May covered cube reorientation on the original Wuji Hand, with a checkpoint that the authors report reached the goal in all 100 simulated trials and 17 of 20 on the real hand, to an average error of about 4 degrees. The September release added the second-generation hand, a 50 Hz reorient variant, and pen spinning, which is a different kind of problem. Rather than reaching a goal pose, the policy tracks recorded reference motions, so the behaviour is shaped by the clips it is given. Five groups of clips ship, each with its own matched checkpoints.
The whole thing assumes Wuji's hardware. The drivers are Wuji's own SDKs, the camera path is written against Hikrobot's industrial camera SDK, and the models describe Wuji's hands. It is a reference implementation for one family of hands rather than a general dexterous-manipulation library.
Why it matters
In-hand reorientation and pen spinning are the classic benchmarks of robot dexterity. They have mostly been shown by large labs, on custom hands, without code a reader could run. Here the maker of a commercially available five-finger hand publishes the complete path, from simulation training to a policy spinning a pen on the real hand, under a permissive licence, on an open simulator. It is a rare complete, runnable example of RL sim-to-real on dexterous hardware, and a template for doing the same on other hands.
Problem it solves
Getting an RL policy for a dexterous hand from simulation onto real hardware normally means writing the environment, tuning the domain randomisation, building a perception rig and a deployment runtime, and discovering every gap between them yourself. wuji-mjlab ships all of those pieces working together for two tasks, with pretrained policies, so an owner of the hand can reproduce the result before changing anything.
Key takeaways
- PPO training in mjlab on MuJoCo, with closed-loop deployment on the physical hand
- Cube Reorient runs on both hand generations; PenSpin tracks reference motions on the Wuji Hand 2, controlling 20 finger joints at 50 Hz
- The release bundles PyTorch and ONNX checkpoints, 40 pen-spinning motion clips with five recipes, and printable jigs, cubes and pens
- Deployment needs only numpy and onnxruntime, split into camera, policy and optional viewer processes over ZeroMQ
- The first release reports 100 of 100 simulated trials and 17 of 20 real trials reaching the goal on the original Wuji Hand
- Apache-2.0; the rest of Wuji's SDKs, hand descriptions and simulation demos are published separately
Who it is for
- Owners of a Wuji Hand or Wuji Hand 2 who want working in-hand manipulation policies
- Researchers studying sim-to-real reinforcement learning for dexterous hands
- mjlab users looking for a complete manipulation task, not just locomotion examples
- Anyone porting in-hand reorientation or motion-tracking policies to a different hand
Limitations
- Built for Wuji's hands: drivers, models and checkpoints target the Wuji Hand and Wuji Hand 2, both commercial products, and support the right hand only
- The camera observer is written against Hikrobot's industrial camera SDK; any other camera needs a code change
- Training requires Linux x86_64, an NVIDIA GPU with CUDA 12.8, and the pixi package manager, which is the only supported installer
- Real-hand success figures are published only for cube reorientation on the first-generation hand; none are given for the Wuji Hand 2 or pen spinning
- Pen spinning follows recorded reference clips, so it reproduces the supplied motions rather than inventing new ones
- Two tasks only: there is no grasping, tool use or bimanual work yet