GenAug: Retargeting behaviors to unseen situations via Generative Augmentation

Authors: Zoey Chen, Sho Kiami, Abhishek Gupta, Vikash Kumar · RSS 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

GenAug uses depth-guided text-to-image diffusion to repaint objects, distractors and backgrounds in a few real demos, so a CLIPort pick-and-place policy works in unseen scenes.

Why it matters

A policy trained from 10 demonstrations in one simple environment can then be used in different real scenes without collecting more robot data. Object masks are needed only for the demonstrations, not at test time.

Problem

Robot learning needs large, diverse datasets, but real-world collection is costly and many robot datasets come from a single setup with a few hours of data. Standard augmentations such as color jitter, blurring and cropping cover low-level appearance changes but not semantic changes such as new distractors, backgrounds or object appearances.

Proposed solution

GenAug takes RGBD demonstrations with object masks and a calibrated camera pose, and uses a pretrained depth-guided text-to-image diffusion model to change object textures within a category, swap object categories by first rendering meshes from Google Scanned Objects and Free3D, add collision-checked distractors, and repaint backgrounds. Because the positions of the pick object and the target stay fixed, the original pick-and-place action labels remain valid; each demonstration is augmented 100 times and the result trains a CLIPort policy.

Key insights

Pick-and-place actions stay valid across large visual changes as long as the approximate positions of the object and target are unchanged, so generated images can reuse the original action labels.
Rendering object meshes before depth-guided generation keeps cross-category replacements 3D-consistent; plain inpainting often produced implausible images (Appendix A).
Copy-paste and random-background augmentations add robustness to new environments but fall short of GenAug on unseen objects, which the authors read as evidence that physically plausible, semantically meaningful scenes matter.
Success rises as the number of augmentations per demonstration grows from 0 to 10, 50 and 100 (Figure 11).

Main results

Real world, 10 tasks with 10 demonstrations each: 85% average success on unseen environments, 52% on unseen objects to place and 45% on unseen objects to pick (Table I, 10 scenes per condition).
Real-world comparison with CLIPort trained without GenAug (Table II, 5 scenes per condition): 80% vs 38% on unseen environments, 54% vs 8% on unseen place objects and 46% vs 10% on unseen pick objects.
The abstract and conclusion report a 40% improvement in generalization to new scenes and objects over training without GenAug.
Simulation, unseen environments with 100 demos (Table III): GenAug scores 77.6 (TransporterNet) and 71.9 (CLIPort), GenAug with depth 91.2 and 73.4, against 9.8 and 14.4 with no augmentation.
GenAug is not the top method in every cell of Table III; for example, Random Background scores 79.1 against GenAug's 77.6 on unseen environments with TransporterNet and 100 demos.

Limitations

GenAug does not augment action labels or reason about material, friction or deformation; it assumes the original action still works and does not collide with added objects (stated by the authors).
It cannot guarantee visual consistency across the frames of a video (stated).
Augmenting one scene takes about 30 seconds, which may be impractical for methods such as on-policy RL (stated).
Failures occur when the object and background share similar colors, when a distractor is very bright or similarly colored, or when the table holds too many objects.
The demonstrations need manually labelled object masks and a calibrated camera pose plus a library of object meshes, and the method was tested only on tabletop pick-and-place with a suction gripper.

Field impact

Finalist for the RSS 2023 Best System Paper Award, with 165 citations. The official project page, genaug.github.io, links a code repository.

Abstract

Robot learning methods have the potential for widespread generalization across tasks, environments, and objects. However, these methods require large diverse datasets that are expensive to collect in real-world robotics settings. For robot learning to generalize, we must be able to leverage sources of data or priors beyond the robot's own experience. In this work, we posit that image-text generative models, which are pre-trained on large corpora of web-scraped data, can serve as such a data source. We show that despite these generative models being trained on largely non-robotics data, they can serve as effective ways to impart priors into the process of robot learning in a way that enables widespread generalization. In particular, we show how pre-trained generative models can serve as effective tools for semantically meaningful data augmentation. By leveraging these pre-trained models for generating appropriate "semantic" data augmentations, we propose a system GenAug that is able to significantly improve policy generalization. We apply GenAug to tabletop manipulation tasks, showing the ability to re-target behavior to novel scenarios, while only requiring marginal amounts of real-world data. We demonstrate the efficacy of this system on a number of object manipulation problems in the real world, showing a 40% improvement in generalization to novel scenes and objects.

Open Source Robotics