An SO-101 Balances a Ping-Pong Ball, Seeing It Only Through the Wrist Camera

SO-101 Open Robot Arm

• Builder: zecice

An SO-101 holds a table-tennis paddle in its gripper and keeps a ping-pong ball balanced on it. The wrist camera is its only sensor for the ball; everything else comes from the arm's own joint readings. The controller was trained in MuJoCo and runs unchanged on the real arm, which the builder shows in a short clip on X: a hand pushes the ball towards the edge of the paddle and the arm brings it back to the middle.

How it works

The pipeline in the repository has four stages. Frames from the wrist camera (640x480, scaled down to 128x96) go to a colour-threshold detector that finds the orange ball. A small MLP estimator, trained in simulation, turns the last 3 detections into the ball's position on the paddle. The velocity is a filtered finite difference of those positions. A controller trained with reinforcement learning (PPO) then outputs 5 joint-position changes at 30 Hz. The paddle is clamped edge-on to the camera, so the camera sees it from the side and the ball near one image column.

What made the jump from simulation work

The builder's sim-to-real lesson is to train the controller with the real system's flaws from the start, a form of domain randomization:

- Control latency randomised over 0 to 4 frames (0 to 133 ms), with the last 3 actions in the observation. - Soft, randomised servo gains, because cheap servos sag and lag. - A noisy ball state like the estimator's output, with its velocity computed by the same finite-difference filter the robot uses. A controller trained on the exact velocity failed completely on velocity derived from the camera.

What did not work is written down too: an end-to-end CNN policy on wrist images, trained by behavior cloning and DAgger from the PPO controller, was fine in simulation but barely saw the real ball. The plain colour detector and the small estimator replaced it.

On the real arm

The hardware is an SO-101 follower arm, a wrist camera, a standard paddle clamped in the gripper and an orange ball. The arm runs through the SO-101 follower and Feetech drivers in LeRobot. The README lists what mattered on the real arm:

- Servo P gain 32 rather than LeRobot's default of 16, at which the arm sags and ignores 4 to 7 degree commands. The run script sets it. - A level home pose: a few degrees of tilt and the ball runs off, because the policy has no integral action. Each arm records its own home pose with an included guide script. - Squeeze the paddle by load (about 10%) and never command the gripper fully closed: a stall trips the Feetech overload protection and the motor drops off the bus until the power is cycled. - Measure your own latency with the included script; the controller is trained for 0 to 4 extra steps of 33 ms.

The README also warns that the arm moves on its own and stays powered after the scripts exit.

Results and what is released

The builder reports 97.5% of episodes balanced for 10 seconds in randomised simulation, 113 of 120 in simulation with the full camera pipeline, and that it balances on the real arm. The trained controller and estimator weights are in the repository, so the whole pipeline can be tried in the simulator without an arm. Training the controller took about 29 million steps and about 2 hours on a laptop (6 CPU cores for the simulation, an RTX A1000 GPU for the network). It was developed on Linux; macOS and Windows are untested.

The code and weights are MIT. The SO-101 model files come from TheRobotStudio's SO-ARM100 repository under Apache-2.0, and the paddle mesh from a table-tennis control project under MIT.

Details

• arm: SO-101 follower
• task: Balancing a ping-pong ball on a table-tennis paddle held in the gripper
• posted: 5 November 2026 (video), repository opened 8 November 2026
• licence: MIT (code and weights); SO-101 model files Apache-2.0; paddle mesh MIT
• sensing: Wrist camera only for the ball, plus the arm's joint readings
• weights: Controller and estimator released in the repository
• training: MuJoCo with randomised latency (0-133 ms), soft servos and noisy ball state
• framework: LeRobot (SO-101 follower, Feetech drivers), MuJoCo 3.17
• controller: PPO policy, 5 joint-position changes at 30 Hz
• difficulty: Advanced
• perception: Colour-threshold detector and a small MLP estimator trained in simulation

Photos and videos

• zecice/so101-pingpong-balance: The simulation, PPO training, detector and estimator, real-arm loop and pretrained weights, with a README covering what made sim-to-real work and what mattered on the real arm.
• The real arm balancing the ball: The builder's clip of the SO-101 keeping the ball on the paddle while a hand pushes it towards the edge.

Links

• Builder: https://github.com/zecice