Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

Authors: Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield · CoRL 2018

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A keypoint network trained only on mixed domain-randomized and photorealistic synthetic images estimates 6-DoF object pose well enough for real robot grasping.

Why it matters

A robot needs no real labelled images to learn a new known object, and because pose comes from PnP rather than direct regression, the network can be used with a different camera without retraining. The paper shows the estimates are accurate enough on a Baxter for top-down grasps, pick-and-place onto another object and handoff from a person.

Problem

Manipulating known objects requires their 3D position and orientation, but labelling 6-DoF pose in real images is labour-intensive, so networks are usually trained on real data closely correlated with the test set and generalize poorly to new cameras and lighting. Synthetic data avoids labelling but suffers from the reality gap, and domain randomization alone had needed fine-tuning on real data to compete.

Proposed solution

DOPE is a six-stage fully convolutional network (the first ten VGG-19 layers plus stages inspired by convolutional pose machines) that outputs belief maps for the 8 projected cuboid vertices and the centroid, plus vector fields pointing from vertices to centroids. Peaks are greedily grouped into object instances and passed to a perspective-n-point (PnP) solver with the camera intrinsics and object dimensions to recover the pose. Training uses about 60k domain-randomized and 60k photorealistic images of YCB objects rendered with the authors' NDDS plugin for Unreal Engine 4.

Key insights

Domain-randomized and photorealistic synthetic data complement each other: the 120k mixed set beat much larger sets of either type alone.
Predicting 2D keypoints and solving PnP keeps camera intrinsics out of the network weights, unlike methods that regress pose directly.
Each extra network stage raises accuracy and costs time, because later stages use larger receptive fields to resolve earlier ambiguity.
Synthetic-only training generalized to a different camera and extreme lighting better than PoseCNN, whose real training images came from the same camera as its test set.
Materials the renderer does not model cause misses: the reflective metal top of the potted meat can was not recognized when it was the only visible part.

Main results

On YCB-Video, DOPE trained only on synthetic data had a higher ADD area under the curve than PoseCNN (trained on synthetic plus real YCB-Video data) for 4 of 5 objects, and did better below 2 cm, about the grasp error limit of the Baxter gripper.
Sugar box, domain-randomized data only: AUC rose from 0.25 at 2k images to 54.22 at 10k and about 60.6 at 100k, peaking at 66.64 with 300k; photorealistic only peaked at 62.94 with 600k; the 120k mixed set reached 77.00.
Mixing ratio: performance was comparable for all networks as long as at least 40% of either dataset was included alongside the full other one.
Stages vs time (including object extraction and PnP): 1 stage gave 46.44 AUC in 57 ms and 6 stages gave 75.69 AUC in 232 ms.
Baxter top-down grasps, 12 trials per object in clutter: cracker box 10, potted meat can 10, mustard bottle 11, sugar box 11, tomato soup can 7 (9 with the can on its side).

Limitations

Severely occluded potted meat cans are often missed because the synthetic data does not model the reflective metal top surface.
Round objects are hard for top-down grasps: the upright soup can succeeded in 7 of 12 trials.
Grasps run without closed-loop servoing; errors come from pose estimation, camera-to-robot calibration and imprecise robot control, with control appearing to play a large role.
The method assumes known rigid objects with known dimensions and known camera intrinsics; the authors list more objects, symmetry handling and closed-loop refinement as future work.
The extreme-lighting comparison with PoseCNN is qualitative only.

Field impact

Cited 778 times. The paper points to video and code on its NVIDIA project page, which links the DOPE code repository; the photorealistic training data was released as the Falling Things (FAT) dataset, and the NDDS data-generation plugin is on GitHub.

Abstract

Using synthetic data for training deep neural networks for robotic manipulation holds the promise of an almost unlimited amount of pre-labeled training data, generated safely out of harm’s way. One of the key challenges of synthetic data, to date, has been to bridge the so-called reality gap, so that networks trained on synthetic data operate correctly when exposed to real-world data. We explore the reality gap in the context of 6-DoF pose estimation of known objects from a single RGB image. We show that for this problem the reality gap can be successfully spanned by a simple combination of domain randomized and photorealistic data. Using synthetic data generated in this manner, we introduce a one-shot deep neural network that is able to perform competitively against a state-of-the-art network trained on a combination of real and synthetic data. To our knowledge, this is the first deep network trained only on synthetic data that is able to achieve state-of-the-art performance on 6-DoF object pose estimation. Our network also generalizes better to novel environments including extreme lighting conditions, for which we show qualitative results. Using this network we demonstrate a real-time system estimating object poses with sufficient accuracy for real-world semantic grasping of known household objects in clutter by a real robot.

Open Source Robotics