R3M: A Universal Visual Representation for Robot Manipulation

Authors: Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, Abhinav Gupta · CoRL 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

R3M pre-trains a frozen ResNet encoder on Ego4D video and captions; behavior cloning on it beats CLIP, MoCo and ImageNet features in 12 sim tasks and on a real Franka with 20 demos.

Why it matters

The encoder is released as pre-trained ResNet18, ResNet34 and ResNet50 weights that load with one call, so a team can train image-based manipulation policies from a few demonstrations without collecting in-domain data for the vision part. The real-robot tasks were learned in a graduate student's apartment from 20 teleoperated demonstrations each.

Problem

Training image-based manipulation policies end to end on data from the target domain is data intensive and generalises poorly. Robotics has no pre-trained visual model, comparable to ImageNet features or BERT, that can be downloaded and reused for any manipulation task, and existing robot datasets cover a limited number of tasks in at most a handful of environments.

Proposed solution

R3M trains a ResNet image encoder on Ego4D, more than 3,500 hours of egocentric human video with language annotations, using three terms: a time-contrastive loss so that frames close in time embed close together, a video-language alignment loss that scores whether the change from a clip's first frame to a later frame matches its caption, and L1 and L2 penalties for sparse, compact embeddings. The encoder is then frozen, and downstream policies are trained by behavior cloning as a two-layer MLP over the image embedding concatenated with robot proprioception.

Key insights

Everyday human video (Ego4D) is usable pre-training data for robot visual representations, even though the embodiment differs from the robot's.
All three training terms contribute; removing video-language alignment causes the largest drop, from 62.4% to 53.2% averaged over environments and from 65.0% to 45.6% on Adroit.
Data alone does not explain the gain: MoCo trained on the same Ego4D frames scores about 10% lower than R3M, and the MVP model about 20% lower, on Franka Kitchen and Adroit.
The sparsity penalty helps on Franka Kitchen and MetaWorld but removing it slightly helps on Adroit, which the authors suspect is because Adroit uses more demonstrations.
R3M's lead holds across every camera viewpoint and demonstration-set size tested, while the ranking between CLIP and MoCo (345) changes with viewpoint.

Main results

Across 12 simulated tasks in MetaWorld, Franka Kitchen and Adroit, R3M reaches about 62% average success, more than 20% above training from scratch and more than 10% above CLIP, supervised ImageNet and MoCo (345) features.
R3M is the best method in all 3 simulation environments and on 11 of 12 tasks.
Ablation (Table 1, all domains): full R3M 62.4%, without crop augmentation 60.4%, without L1 penalty 59.4%, without language alignment 53.2%.
Data versus objective (Table 2): Franka Kitchen 53.1% for R3M, 42.0% for MoCo trained on Ego4D, 27.0% for MVP; Adroit 65.0%, 54.9% and 51.4%.
Real Franka Emika Panda, 20 demonstrations and 10 trials per task: R3M averages 56% against 24% for CLIP (closing drawer 80% vs 70%, mask in dresser 30% vs 10%, lettuce in pan 60% vs 0%, pushing mug 70% vs 40%, folding towel 40% vs 0%).

Limitations

Evaluation is limited to imitation learning, specifically behavior cloning with a small number of demonstrations; a good pre-trained representation for reinforcement learning may differ from one for imitation.
R3M provides only a single-frame state representation; using it, or its language-grounding module, for reward learning or task specification is left for future work.
The MoCo (345) baseline is set up differently from Parisi et al. (proprioception, frame stacking), so its numbers are not directly comparable across the two works.
The real-robot comparison covers only R3M and CLIP, with 10 trials per task.
Simulation success rates are the best value reached across evaluations every 1,000 training steps, averaged over seeds, viewpoints and demonstration-set sizes.

Field impact

Cited 541 times. The authors release pre-trained R3M models with ResNet18, ResNet34 and ResNet50 backbones together with the training code.

Abstract

We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks. Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations. The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning. Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and MoCo. Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations.

Open Source Robotics