Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations

Authors: Siddhant Haldar, Jyothish Pari, Anant Rai, Lerrel Pinto · RSS 2023

Licences

Open Access
Paper: CC0 1.0 Universal

Everything below draws on the full paper, which its authors released under CC0 1.0 Universal; its figure is credited to them.

TL;DR

FISH adds a learned residual to a fixed open-loop or nearest-neighbour base policy, rewarded by optimal-transport matching to 1–3 demos, and adapts to unseen object positions in ≤20 minutes.

Why it matters

The only task input is under a minute of demonstrations: no state estimation, no hand-built reward and no large demonstration set, with online training capped at 20 minutes. Because only the residual is learned, exploration can be limited to chosen action dimensions, which the authors show avoided the unsafe finger collisions and poses seen when ROT explored all dimensions on the Allegro Hand.

Problem

Visual imitation policies need many demonstrations to cover variations such as object position; the paper cites MIME at 400 demonstrations per task and robomimic at 200. Adapting an offline policy with online RL can reduce this, but RL needs a reward, and task-specific rewards are hard to build when object states cannot be estimated.

Proposed solution

FISH works in two phases. First, a weak non-parametric base policy (open-loop replay or VINN nearest-neighbour retrieval) is fitted to about one minute of demonstrations (1–3 trajectories). Second, a residual policy trained with n-step DDPG adds corrective offsets to the base action, rewarded by a Sinkhorn optimal-transport match between the robot's camera observations and the demonstration, computed with a frozen encoder taken from behavior cloning.

Key insights

Keeping the base policy fixed and learning only residual offsets beat finetuning a BC policy online (ROT) on every task in Table I.
Non-parametric base policies (VINN, open-loop) worked better than parametric BC in this low-data regime; BC performed poorly on the Allegro Hand's 16-dimensional action space.
Encoders trained by behavior cloning on the few in-domain demonstrations gave better results than ImageNet, R3M or MVP encoders pretrained on large out-of-domain data.
Freezing the encoder (so OT rewards stay stationary) and conditioning the residual on the base action are both needed; dropping either lowered success (Table V).
Restricting residual exploration to chosen action axes improved sample efficiency, while adaptively regularizing the offsets toward zero hurt performance.

Main results

Average success rate of 93% across 9 real-world tasks on 3 robots, around 3.8× higher than prior methods (abstract); the introduction words the same figure as a '93% improvement in success rate' given 20 minutes of online interaction.
Table I: 1.0 on door opening, drawer opening and light switching (Stretch, 5 trials each) and on cube flipping and bottle-cap spinning (Allegro Hand); 0.8 on dollar-bill picking, 0.9 on peg-in-cup and bagel flipping, 0.8 on key insertion (10 trials each).
The best baseline on any single task in Table I reaches at most 0.6 (ROT on light switching and key insertion); on door opening, drawer opening, cube flipping and bottle-cap spinning no baseline exceeds 0.2.
Without adaptive offset regularization, success was 0.9 on bagel flipping and 0.8 on dollar-bill picking, against 0.4 and 0.4 with it (Table II).
Replacing the BC encoder with pretrained encoders dropped bagel flipping from 0.9 to 0.0 (ImageNet), 0.0 (R3M) and 0.3 (MVP), and dollar-bill picking from 0.8 to 0.0, 0.1 and 0.0 (Table IV).
On new objects after 5 minutes of online learning, success ranged from 0.9 to 0.0 for breads and from 0.8 to 0.0 for bills and cards; a soft flatbread and a thick wallet failed completely (Fig. 8).

Limitations

OT rewards align the robot with the demonstrations, so the method relies on the demonstrator being an expert (stated by the authors).
Vision-only inputs make precise tasks with weak visual signals hard, such as finding a keyhole that spans a tiny part of the image; the authors suggest tactile sensing (stated).
The residual policy is randomly initialized; the authors note pretraining it might help harder tasks that need more precise control (stated).
Success falls as objects differ more from the demonstrated one, down to complete failure on a flatbread and a wallet.
Each task is evaluated on 10 object positions (xArm, Allegro Hand) or 5 robot initializations (Stretch).

Field impact

Winner of the RSS 2023 Best Student Paper Award, with 113 citations. The authors release code and robot videos through fast-imitation.github.io, and the project page links the code repository.

Abstract

While imitation learning provides us with an efficient toolkit to train robots, learning skills that are robust to environment variations remains a significant challenge. Current approaches address this challenge by relying either on large amounts of demonstrations that span environment variations or on handcrafted reward functions that require state estimates. Both directions are not scalable to fast imitation. In this work, we present Fast Imitation of Skills from Humans (FISH), a new imitation learning approach that can learn robust visual skills with less than a minute of human demonstrations. Given a weak base-policy trained by offline imitation of demonstrations, FISH computes rewards that correspond to the "match" between the robot's behavior and the demonstrations. These rewards are then used to adaptively update a residual policy that adds on to the base-policy. Across all tasks, FISH requires at most twenty minutes of interactive learning to imitate demonstrations on object configurations that were not seen in the demonstrations. Importantly, FISH is constructed to be versatile, which allows it to be used across robot morphologies (e.g. xArm, Allegro, Stretch) and camera configurations (e.g. third-person, eye-in-hand). Our experimental evaluations on 9 different tasks show that FISH achieves an average success rate of 93%, which is around 3.8× higher than prior state-of-the-art methods.

Open Source Robotics