Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation

Authors: Peter R. Florence, Lucas Manuelli, Russ Tedrake · CoRL 2018

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A robot scans objects with a wrist RGBD camera, self-supervises per-pixel descriptors from 3D reconstructions, then grasps user-chosen points across poses, deformations and instances.

Why it matters

Dense descriptors let a robot refer to a specific point on an object, such as a shoe's heel or a toy's ear, across new poses, deformations and other instances of a class. The robot collects and labels its own training data, and a new object can be learned in about 20 minutes.

Problem

General grasping methods pick any graspable item but keep no consistent object representation, while task-specific learning may need many trials to generalize across object configurations or tasks. Instance segmentation gives only whole-object masks and says nothing about the structure of the object. The authors want a task-agnostic representation for rigid and non-rigid objects that uses 3D vision priors and is learned entirely from self-supervision.

Proposed solution

A Kuka arm with a wrist-mounted RGBD camera scans each static scene, and TSDF fusion with camera poses from forward kinematics gives a dense 3D reconstruction that supplies pixel correspondences between images for a pixelwise contrastive loss on a ResNet-34 fully convolutional network. Object masks from 3D change detection enable masked match sampling and background domain randomization, hard-negative scaling keeps the non-match loss informative, and cross-object loss and synthetic multi-object scenes make descriptors distinct across objects. At run time a user clicks one pixel in a reference image, the robot finds the best descriptor match in a new scene and grasps the corresponding 3D point.

Key insights

Static-scene 3D reconstructions from a calibrated robot provide dense pixel correspondences with no human labels.
Static-scene data association alone did not give consistent object descriptors; object masking, background and orientation randomization, and hard-negative scaling were needed.
Training only within scenes makes different objects overlap in descriptor space; cross-object loss separates them, at a cost in correspondence precision for low-dimensional descriptors that slightly larger dimensions recover.
Trained jointly on similar instances, descriptors generalize across a class (hats, shoes, mugs), while the 'specific' training mode separates even very similar instances.
Projecting descriptors to the unit sphere gave substantial gains when going from D = 4 to D = 8, though it was not used in the hardware experiments.

Main results

Descriptors were learned for 47 objects across 275 scenes, including 3 classes (8 hats, 15 shoes, 15 mugs) and 9 other objects.
Training runs 3500 steps at about 0.22 s per step (about 13 minutes) on a single Nvidia 1080 Ti or Titan Xp; with scene collection a new object takes about 20 minutes.
With the standard single-object procedure, in 93% of image pairs the best match lay within 13% of the image diagonal of the true match; it outperformed the authors' implementation of Schmidt et al.'s procedure, and object masking and hard-negative scaling each gave measurable gains.
Hat descriptors trained on only 6 instances generalized to unseen hats, including a blue hat, a colour never seen in training.
Robot demonstrations: grasping a caterpillar toy's tail or right ear across configurations, a class-general point on unseen shoes after training on 4 shoes, and one shoe's heel in a cluttered pile of shoes.
Each scan takes about 70 s and captures about 2,100 RGBD frames at 30 Hz, downsampled to about 315 images per scene.

Limitations

With multiple objects, a human must still swap objects for the robot and indicate which scenes correspond to which object.
Sufficiently different objects do not share descriptors: the anthropomorphic toys Baymax and Starbot did not get general descriptors.
Descriptors cannot guide grasps before learning has happened and do not constrain gripper orientation, so grasps rely on a simple geometric point-cloud planner.
Cross-object loss lowers correspondence precision for low-dimensional descriptor spaces in multi-object networks.
The grasp demonstrations are shown qualitatively; the paper reports no grasp success rates.

Field impact

Winner of the CoRL 2018 Best Paper Award, with 321 citations. Code, data and video are released at github.com/RobotLocomotion/pytorch-dense-correspondence.

Abstract

What is the right object representation for manipulation? We would like robots to visually perceive scenes and learn an understanding of the objects in them that (i) is task-agnostic and can be used as a building block for a variety of manipulation tasks, (ii) is generally applicable to both rigid and non-rigid objects, (iii) takes advantage of the strong priors provided by 3D vision, and (iv) is entirely learned from self-supervision. This is hard to achieve with previous methods: much recent work in grasping does not extend to grasping specific objects or other tasks, whereas task-specific learning may require many trials to generalize well across object configurations or other tasks. In this paper we present Dense Object Nets, which build on recent developments in self-supervised dense descriptor learning, as a consistent object representation for visual understanding and manipulation. We demonstrate they can be trained quickly (approximately 20 minutes) for a wide variety of previously unseen and potentially non-rigid objects. We additionally present novel contributions to enable multi-object descriptor learning, and show that by modifying our training procedure, we can either acquire descriptors which generalize across classes of objects, or descriptors that are distinct for each object instance. Finally, we demonstrate the novel application of learned dense descriptors to robotic manipulation. We demonstrate grasping of specific points on an object across potentially deformed object configurations, and demonstrate using class general descriptors to transfer specific grasps across objects in a class.

Open Source Robotics