XIRL: Cross-embodiment Inverse Reinforcement Learning

Authors: Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, Debidatta Dwibedi · CoRL 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Temporal cycle-consistency embeddings learned from videos of differently embodied demonstrators give a goal-distance reward that trains RL policies on unseen embodiments.

Why it matters

The reward needs only videos: no action labels, no paired frames and no environment instrumentation beyond a camera. In the paper's experiments the same learned reward trained agents whose bodies were absent from the demonstration set, which is the situation when teaching a robot from human video.

Problem

Robots could learn tasks from third-person videos, such as humans demonstrating them, but the demonstrator's body differs from the robot's, so each solves the task with a different strategy and trajectory. Earlier reward-learning methods needed a reference trajectory, a time-index alignment, or human-labelled frame correspondences between demonstrator and learner.

Proposed solution

XIRL trains a ResNet-18 image encoder with temporal cycle-consistency (TCC) on observation-only videos pooled from several embodiments, without embodiment labels. The reward is a sigmoid of the negative, temperature-scaled squared distance between the current frame's embedding and the mean embedding of the demonstrations' final frames, and a Soft Actor-Critic policy is trained on that reward for the new embodiment.

Key insights

TCC embeddings encode task progress, so the distance to a single goal embedding gives a dense reward without a reference trajectory or time-index alignment.
Pooling demonstrations from more embodiments, with the total held at 1000 demonstrations, sped up RL on a held-out embodiment.
Different bodies solve the same task very differently: in X-MAGICAL the mean demonstration length ranges from 48.0 steps (long-stick) to 110.8 (gripper).
The learned reward gives partial credit for partial progress, e.g. for moving towards a drawer before colliding with it, where the sparse environment reward stays at zero.

Main results

Same-embodiment X-MAGICAL sweeping: XIRL was at least as sample-efficient as the other learned rewards and more sample-efficient than the environment reward in 3 of 4 embodiments.
Cross-embodiment, each of the 4 agents held out with the encoder trained on the other 3: XIRL generalised to the new agent significantly better than TCN, LIFS and goal-classifier rewards (5 seeds, 50 evaluation episodes each).
Real human videos from the RLV dataset training a simulated Sawyer arm to push a block: XIRL plus the sparse reward was more sample-efficient than RLV and than the sparse reward alone, without paired-frame labels or robot-domain data in reward learning.
X-REAL: 93 real videos of 9 end-effectors (human hands, ski gloves, kitchen tongs, lobster-hand costume, tweezers, clamps, a grabber reacher) moving five pens into two cups; for One Hand 5 Fingers an incomplete attempt received roughly half the reward of a successful one.
X-MAGICAL benchmark: 4 simulated embodiments with 1000 expert demonstrations each.

Limitations

No policy was trained on a real robot; the authors state they have yet to show policy learning on a real robot.
X-MAGICAL is a 2D sweeping task built on PyMunk, which the authors describe as a highly simplified version of the real-world scenario.
Demonstrations contain only successful trajectories, so a poorly generalising encoder may define a different task than intended; the paper suggests combining the learned reward with a sparse task reward.
On the X-REAL real-world dataset the paper shows learned reward curves only, not policy learning.
Most results are given as learning curves rather than tabulated success rates, so few exact numbers are reported.

Field impact

Finalist for the CoRL 2021 Best Paper Award, with 182 citations as given. The paper introduces the X-MAGICAL benchmark and the X-REAL dataset; the licensed v1 copy says code and datasets will be made available but gives no link.

Abstract

We investigate the visual cross-embodiment imitation setting, in which agents learn policies from videos of other agents (such as humans) demonstrating the same task, but with stark differences in their embodiments – shape, actions, end-effector dynamics, etc. In this work, we demonstrate that it is possible to automatically discover and learn vision-based reward functions from cross-embodiment demonstration videos that are robust to these differences. Specifically, we present a self-supervised method for Cross-embodiment Inverse Reinforcement Learning (XIRL) that leverages temporal cycle-consistency constraints to learn deep visual embeddings that capture task progression from offline videos of demonstrations across multiple expert agents, each performing the same task differently due to embodiment differences. Prior to our work, producing rewards from self-supervised embeddings has typically required alignment with a reference trajectory, which may be difficult to acquire. We show empirically that if the embeddings are aware of task-progress, simply taking the negative distance between the current state and goal state in the learned embedding space is useful as a reward for training policies with reinforcement learning. We find our learned reward function not only works for embodiments seen during training, but also generalizes to entirely new embodiments. We also find that XIRL policies are more sample efficient than baselines, and in some cases exceed the sample efficiency of the same agent trained with ground truth sparse rewards.

Open Source Robotics