Disentangled Relational Representations for Explaining and Learning from Demonstration

Authors: Yordan Hristov, Daniel Angelov, Michael Burke, Alex Lascarides, Subramanian Ramamoorthy · CoRL 2019

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A β-VAE with auxiliary label losses learns spatial-relation embeddings that turn table-top demonstrations into symbolic plans and PR2 end-effector poses.

Why it matters

A user can teach a robot a spatial vocabulary from partly labelled data, and the robot then explains new demonstrations in that same vocabulary, so its plan can be read and checked by a person. The same embeddings feed a pose regressor, which links the taught symbols to arm motion on a PR2.

Problem

Learning from demonstration depends on an inductive bias in the structure of rewards and policies, and people who are not robotics experts find it easier to demonstrate a task than to specify it. A robot taught with language must ground abstract concepts such as 'left of' or 'on' in its high-dimensional sensor stream. Prior relational models need large fully labelled datasets and act as black boxes, or assume a known environment model, a deterministic mapping to symbols and full plans.

Proposed solution

A Siamese β-VAE encodes masked RGBD images of a target and a reference object into object embeddings, and a variational operator maps each pair into a low-dimensional relational space with one axis per concept group (left/right, front/behind, above/below, close/far, on/off, in/out). Reconstruction, KL and cross-entropy terms on weak, partly missing labels align these axes with human notions. At test time, likelihood-ratio tests on the embedding traces of demonstrations recover which object moves when and how its relations change, the steps found in all demonstrations form the plan, and an MLP regresses a 6-DoF end-effector pose that is passed to the MoveIt! motion planner.

Key insights

Weak labels with an explicit 'unknown' case are enough to align latent axes with human spatial concepts; unlabelled pairs still contribute through the reconstruction loss.
Combining masked reconstruction with object-label classification gave the most usable relational space for segmenting demonstrations; models without the object-label loss produced relational embeddings packed tightly around zero, which the authors link to their weaker segmentation.
Likelihood-ratio tests between a 'moving' and a 'stationary' Gaussian on consecutive embeddings are enough to find which object moves when.
Keeping only the steps found in every demonstration removes incidental moves, such as passing behind an object on the way from its left to its right.
The pose regressor is most accurate on the axes that matter for each task: position for placing on or in, roll and pitch for making two cups face each other.

Main results

Plan segmentation on repetitive BlocksWorld demonstrations: the full model (reconstruction + object-label loss) scored 0.80 left-right, 0.88 front-behind, 0.91 below-above, 0.86 far-close and 0.76 off-on, against 0.50, 0.64, 0.54, 0.56 and 0.49 for the model with neither term.
Exception: on the out-in group the full model scored 0.56, below all three ablations (0.66, 0.62 and 0.78).
Chained demonstrations, where a single object moves: all models reached accuracy 1 on C-shape and jump-over and about 0.74 on off-on-off.
Mean edit distance of inferred plans falls toward the ground truth as more demonstrations (up to 8) are used; some curves do not converge because of object occlusions in some demonstrations.
Data: 1,000 synthetic BlocksWorld scenes with 4 objects each (72,000 relations, 40% unlabelled) and, on the PR2, 3 teleoperated tasks × 20 demonstrations giving 2,400 labelled and 6,000 unlabelled object pairs; pose prediction was tested on 10 further demonstrations per task.

Limitations

Task essence extraction only works for deterministic tasks, where a single sequence of actions achieves the goal.
Grasping is not addressed; the robot is assumed to already hold the object to be moved.
Segmentation is not learned: masks come from a pre-trained Mask R-CNN on the robot and from the simulator state in BlocksWorld, and instructions must follow a predefined template for the parser.
Object occlusions in some demonstrations keep the inferred plans from converging to the ground truth.
Evaluation covers three simple table-top tasks; compositional tasks and modifiers such as 'more to the left' are left for future work.

Field impact

Finalist for the CoRL 2019 Best Paper Award, with 28 citations. The official project page links the experiment code, the data-generation code and the photorealistic CLEVR-based training data.

Abstract

Learning from demonstration is an effective method for human users to instruct desired robot behaviour. However, for most non-trivial tasks of practical interest, efficient learning from demonstration depends crucially on inductive bias in the chosen structure for rewards/costs and policies. We address the case where this inductive bias comes from an exchange with a human user. We propose a method in which a learning agent utilizes the information bottleneck layer of a high-parameter variational neural model, with auxiliary loss terms, in order to ground abstract concepts such as spatial relations. The concepts are referred to in natural language instructions and are manifested in the high-dimensional sensory input stream the agent receives from the world. We evaluate the properties of the latent space of the learned model in a photorealistic synthetic environment and particularly focus on examining its usability for downstream tasks. Additionally, through a series of controlled table-top manipulation experiments, we demonstrate that the learned manifold can be used to ground demonstrations as symbolic plans, which can then be executed on a PR2 robot.

Open Source Robotics