You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration

Authors: Bowen Wen, Wenzhao Lian, Kostas Bekris, Stefan Schaal · RSS 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A canonical shape space trained only in simulation maps one demo video onto new objects; tracked closed-loop cloning reaches 100% on battery standing and 82.9% on 0.5 mm gear insertion.

Why it matters

A new assembly skill can be taught with one video plus CAD models of training instances, without real-world training data or keypoint annotation, and it carries over to new instances bought from retailers. The comparison with and without tracking shows how much closed-loop object-pose feedback matters in tight-tolerance insertion.

Problem

Category-level manipulation methods that transfer a skill across object instances usually need manually specified semantic keypoints and costly real-world data collection, and they treat the grasped object as fixed to the gripper and execute open loop. This breaks down in long-horizon, contact-rich tasks such as inserting a battery against a spring, where the intermediate action sequence matters and the object moves in the hand.

Proposed solution

A NUNOCS Net (a PointNet-like network) trained only on synthetic Blender data maps a partial point cloud into a category-level canonical space with 3D non-uniform scale, giving dense correspondence between the demonstrated object and a new one. One RGBD demonstration video is parsed with the model-free BundleTrack 6 DoF tracker into an object pose trajectory relative to the receptacle, reprojected onto the new object, and executed as an RRT* collision-free move to a keypose 5 cm from the receptacle followed by category-level behavior cloning (CatBC). CatBC follows the target trajectory with Jacobian-based joint updates and tracker feedback at 10 Hz, anchored to a frame chosen by a local attention mechanism.

Key insights

Expressing the demonstration as the object's pose trajectory relative to the receptacle separates the skill from the demonstrator (human hand or teleoperated robot) and from the scene layout.
Closed-loop tracking of the in-hand object matters most in contact-rich tasks: success rose from 37.78% to 82.22% in battery assembly and from 42.9% to 100% in gear insertion at the loosest tolerance when tracking was turned on.
Dense 3D correspondence from a canonical space trained in simulation was more precise than 2D dense descriptors (DON) or semantic keypoints (KPAM, KPAM 2.0) trained on real data; even without tracking it beat all baselines.
Splitting execution into collision-free transport and last-inch cloning lets the robot handle obstacles not seen in the demonstration and reproduce only the part of the video that decides success.
Different strategies can be taught by re-demonstrating: an anchor-and-pivot demonstration gave higher gear-insertion success than plain top-down insertion.

Main results

Battery standing (9 test batteries, 5 trials each): 100% for the full method, 80.0% without tracking, and at most 60.0% for the baselines (KPAM 2.0 with demonstration-based cloning).
Battery assembly against a spring: 82.22% for the full method, 37.78% without tracking, at most 24.44% for the baselines, and 0% for KPAM, KPAM 2.0 and DON without cloning.
Gear insertion (7 test gears): 100% at 5 mm tolerance (3 mm for one gear), 82.9% at 0.5 mm and 28.6% at 0.1 mm; the best baseline reached 31.4% at 5 mm, all six baselines scored 0% at 0.5 mm, and the no-tracking variant scored 5.7% at 0.5 mm.
1,560 real-world trials were run across the method and 7 baselines; the tracker runs at the camera rate of 10 Hz.
The NUNOCS Net used 100,000 synthetic data points per category, while the KPAM and KPAM 2.0 baselines were trained on 90,000 to 98,340 real-world data points per category with keypoint annotations.

Limitations

Vision is the only sensing modality; the authors expect force or tactile sensing to improve accuracy in high-precision tasks.
Only rigid objects are handled; articulated or deformable objects such as cables need other category-level representations.
Reported failure modes: in battery assembly the spring force can push the battery out of the gripper, and in anchor-and-pivot gear insertion the gear can rotate in the hand until no collision-free inverse-kinematics solution exists.
At 0.1 mm gear-shaft tolerance success falls to 28.6%.
The method assumes a grasp that does not obstruct the task (grasp planning is outside its scope), and the receptacles are built from geometric primitives and located with RANSAC.

Field impact

Finalist for the RSS 2022 Best Paper Award, and cited 145 times. The paper does not state a release of its own code or data.

Abstract

Promising results have been achieved recently in category-level manipulation that generalizes across object instances. Nevertheless, it often requires expensive real-world data collection and manual specification of semantic keypoints for each object category and task. Additionally, coarse keypoint predictions and ignoring intermediate action sequences hinder adoption in complex manipulation tasks beyond pick-and-place. This work proposes a novel, category-level manipulation framework that leverages an object-centric, category-level representation and model-free 6 DoF motion tracking. The canonical object representation is learned solely in simulation and then used to parse a category-level, task trajectory from a single demonstration video. The demonstration is reprojected to a target trajectory tailored to a novel object via the canonical representation. During execution, the manipulation horizon is decomposed into long-range, collision-free motion and last-inch manipulation. For the latter part, a category-level behavior cloning (CatBC) method leverages motion tracking to perform closed-loop control. CatBC follows the target trajectory, projected from the demonstration and anchored to a dynamically selected category-level coordinate frame. The frame is automatically selected along the manipulation horizon by a local attention mechanism. This framework allows to teach different manipulation strategies by solely providing a single demonstration, without complicated manual programming. Extensive experiments demonstrate its efficacy in a range of challenging industrial tasks in high-precision assembly, which involve learning complex, long-horizon policies. The process exhibits robustness against uncertainty due to dynamics as well as generalization across object instances and scene configurations.

Open Source Robotics