FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects

Authors: Ben Eisner, Harry Zhang, David Held · RSS 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A network predicts how each point of an articulated object would move; the robot grasps the point with the largest motion and follows it, reaching 64% real-world success with no real training data.

Why it matters

The representation needs no knowledge of the object's kinematic tree, so a single model trained only on simulated point clouds opened objects from categories it never saw and transferred to a real Sawyer arm without fine-tuning. For practitioners, it separates perception (where and how parts move) from a simple analytical control rule, which keeps the policy closed-loop and easy to inspect.

Problem

Robots in homes must open doors, drawers, lids and similar articulated objects whose geometry and kinematics vary widely within and across categories. Methods that estimate articulation parameters such as joint axes usually rely on class-specific knowledge of the object's structure, so they cannot be applied to new categories.

Proposed solution

The authors define 3D Articulation Flow: for every point in a point cloud, the normalized direction it would move if its part were opened slightly. ArtFlowNet, a PointNet++-based network trained with supervised learning on simulated PartNet-Mobility objects, predicts this flow from a single point cloud. FlowBot3D then grasps the point with the largest predicted flow using a suction gripper (subject to edge and curvature filters) and repeatedly re-predicts the flow and moves the gripper along the largest visible flow vector near the contact until the part is open.

Key insights

For a prismatic joint the best pulling direction is along the joint axis at any point; for a revolute joint it is tangential at the point farthest from the axis, so the point with the largest articulation flow is the ideal grasp.
Per-point flow is hierarchy-free and bounded within the unit sphere, so it is invariant to translation and scaling and can be learned without class-specific kinematic models.
Behaviour cloning and DAgger policies performed poorly even when given ground-truth flow as input, while the analytical flow-following policy did well.
Point clouds rendered in simulation have little domain gap from real depth data, which the authors credit for direct sim-to-real transfer; viewpoint augmentation in training helped transfer but changed little in simulation.

Main results

Simulation (ManiSkill with PartNet-Mobility, success = over 90% of the range of motion): FlowBot3D 0.77 average success on unseen instances of the training categories and 0.69 on 10 unseen categories; an oracle using ground-truth flow reached 0.92 and 0.82.
Best behaviour-cloning baseline (DAgger Oracle + F) reached 0.33 success on training categories and 0.16 on unseen ones; the UMPNet-style baseline reached 0.52 and 0.45.
Average normalized distance to the open state: 0.12 (training categories) and 0.15 (unseen categories) for FlowBot3D vs 0.29 and 0.36 for the UMPNet-style baseline.
Real world, 14 household objects and 70 trials on a Sawyer arm: 45/70 (64.3%) overall success, 64/70 (91.4%) contact success and 45/64 (70.3%) motion success after contact.
The real-world DAgger baseline, using the same contact selection, succeeded in 10/70 trials (14.3%), with average distance 0.73 vs 0.22 for FlowBot3D.

Limitations

Most real-world failures came from flow prediction errors, often when the robot occluded much of the scene or failed to detect an articulation (for example on the real oven), and from suction contact failures on points where a seal could not form.
The authors state that a more compliant arm controller and a stronger contact-quality prediction module are needed to remove many failure modes; online correction of wrong predictions is left to future work.
The method covers objects with 1-DoF revolute or prismatic joints and a fixed base; socket joints, free-floating objects and deformable objects are excluded.
A real suction gripper cannot seal on edges, handles or high-curvature spots, so grasp points are chosen with hand-tuned edge, curvature and distance heuristics instead of the ideal maximum-flow point.
Real test objects had to fit the workspace, offer a suction-friendly surface and not be too dark or reflective for the depth camera.

Field impact

The paper was a finalist for the RSS 2022 Best Paper Award and has 168 citations. The authors release code, data and supplementary materials through the project website.

Abstract

We explore a novel method to perceive and manipulate 3D articulated objects that generalizes to enable a robot to articulate unseen classes of objects. We propose a vision-based system that learns to predict the potential motions of the parts of a variety of articulated objects to guide downstream motion planning of the system to articulate the objects. To predict the object motions, we train a neural network to output a dense vector field representing the point-wise motion direction of the points in the point cloud under articulation. We then deploy an analytical motion planner based on this vector field to achieve a policy that yields maximum articulation. We train the vision system entirely in simulation, and we demonstrate the capability of our system to generalize to unseen object instances and novel categories in both simulation and the real world, deploying our policy on a Sawyer robot with no finetuning. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments.

Open Source Robotics