SORNet: Spatial Object-Centric Representations for Sequential Manipulation

Authors: Wentao Yuan, Chris Paxton, Karthik Desingh, Dieter Fox · CoRL 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

SORNet embeds each object in an RGB image given one canonical view of it; small readout MLPs turn the embeddings into skill preconditions and 3D directions for unseen objects.

Why it matters

The predicted predicates feed a symbolic task and motion planner, so the same network can serve new tasks and new objects without retraining or fine-tuning. The authors use predicates predicted from a real robot's camera image to populate a planner's state for open-loop execution, and use direction predictions for visual servoing to a target object.

Problem

Sequential manipulation needs to know which spatial relations hold between objects, for example whether a block's top is clear, before a planner can choose the next skill. Explicit state estimators are hard to generalise to arbitrary collections of objects, especially objects in contact, and end-to-end networks stay tied to the task they were trained on: a network trained to stack blocks cannot simply unstack them.

Proposed solution

SORNet is a Vision Transformer that takes RGB context patches of the scene plus canonical views of the objects of interest as query tokens, with attention masked so that the queries read only from the scene, and outputs one embedding per query object. Two-layer MLP readout networks map single or paired embeddings to unary and binary predicates or to continuous 3D directions, with parameters independent of the number of objects; training uses Leonardo, a new simulated dataset of a Franka Panda stacking blocks, labelled with ground-truth predicates.

Key insights

Conditioning on canonical object views, which need not match the object's appearance in the scene, lets the model pick out unseen objects under changes in lighting, pose and occlusion without retraining.
Masking attention among the query tokens and from context patches to queries makes the outputs permutation equivariant, so any number of objects can be queried at inference.
Embeddings trained only with logical predicate labels still carry continuous spatial information: a regressor on frozen embeddings predicts 3D directions better than baselines trained with pose supervision.
Baselines without object conditioning fail zero-shot on unseen objects and remain far behind even after 100-shot fine-tuning.
Aggregating three camera views helps with occlusion, and adding the gripper state improves predicates tied to the gripper, such as has-object.

Main results

CLEVR-CoGenT zero-shot relation accuracy: 99.006% on valA and 98.222% on valB, versus 84.950% / 59.627% for MDETR and 97.944% / 98.052% for MDETR-oracle, which saw all objects in training.
Leonardo predicate classification on unseen objects (52 predicates): SORNet M-View (G) reaches 98.9% accuracy, 63.6% all-match accuracy and 89.5 F1; the best 100-shot baseline reaches 92.8%, 4.1% and 0.0.
Scenes with 5 and 6 objects (trained on 4): 98.5% and 98.0% accuracy, with all-match falling to 45.5% and 29.7% as predicates grow to 70 and 102.
30 real-world robot images: 96.3% accuracy, 26.7% all-match accuracy, 76.5 F1.
Skill executability accuracy: 76.3% for SORNet M-View (G) versus about 27% for ResNet18 and ViT baselines, which score 0% on grasp, align, place and lift.
Relative 3D direction regression (Euclidean error on unit vectors, 1000 training examples): 0.1458 object-object and 0.1777 end-effector-object for SORNet (pretrained, multi-view), versus 0.4308 / 0.3251 for ResNet18 and 0.6145 / 0.4960 for pretrained CLIP-ViT.

Limitations

Readout networks handle unary and binary relations only; relations involving more than two objects are left for future work.
Spatial relations are predicted from a single RGB frame; temporal reasoning is not addressed.
On real images the model makes mistakes in situations never seen in training, such as one block stacked between two blocks, and all-match accuracy on the 30 real images is 26.7%.
All-match accuracy falls as objects are added (29.7% with 6 objects) because the number of binary predicates grows quadratically.
Training data covers a single task, stacking 4 blocks; the real-robot planning demonstration runs open loop and is shown only in the supplementary video.

Field impact

CoRL 2021 Best System Paper Award finalist, with 82 citations. The paper contributes the Leonardo dataset of RGB observations labelled with spatial predicates, and the official project page links the code (github.com/wentaoyuan/sornet).

Abstract

Sequential manipulation tasks require a robot to perceive the state of an environment and plan a sequence of actions leading to a desired goal state, where the ability to reason about spatial relations among object entities from raw sensor inputs is crucial. Prior works relying on explicit state estimation or end-to-end learning struggle with novel objects or new tasks. In this work, we propose SORNet (Spatial Object-Centric Representation Network), which extracts object-centric representations from RGB images conditioned on canonical views of the objects of interest. We show that the object embeddings learned by SORNet generalize zero-shot to unseen object entities on three spatial reasoning tasks: spatial relation classification, skill precondition classification and relative direction regression, significantly outperforming baselines. Further, we present real-world robotic experiments demonstrating the usage of the learned object embeddings in task planning for sequential manipulation.

Open Source Robotics