Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Plans

Authors: Homanga Bharadhwaj, Abhinav Gupta, Vikash Kumar, Shubham Tulsiani · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

HOPMan predicts future hand and object masks from a scene and goal image using web video, then translates that plan into Franka Panda actions, covering 16 skills and 40 objects zero-shot.

Why it matters

Most of the knowledge about how hands and objects move comes from passive web video, so the robot-specific data stays small: the authors collected 600 paired trajectories in about 3 days. One goal-conditioned policy is tested on unseen object instances, unseen object-skill pairs, unseen object categories and unseen skills, including on a mobile base in unmodified kitchens and offices.

Problem

Collecting robot interaction data for every task, object and skill does not scale, and recent broad-capability systems needed years of robot data while remaining largely limited to picking, placing and pushing. Methods that use human video for visual representations or affordances still need many robot demonstrations or deployment-time training, and others learn a separate policy per task.

Proposed solution

The policy is factorized into a goal-conditioned diffusion model that predicts seven future hand and object mask frames from a current image and a goal image, trained on about 150,000 two-second clips from EPIC-KITCHENS and Ego4D, and a transformer translation policy that maps the predicted masks and current observations to 8-D actions (7 joint positions plus gripper open/close). The translation model is trained on about 400 in-lab and 200 in-the-wild paired human-robot trajectories, plus about 1000 robot-only trajectories in which the robot is inpainted out and a human hand inpainted in.

Key insights

Semantic masks of the hand and the manipulated object form a compact plan space that drops task-irrelevant scene detail and can be learned from uncurated human video.
Splitting the policy into plan prediction and plan-to-action translation lets the data-hungry part learn from web video while the robot-specific part needs only small paired data.
Hallucinated pairs (robot videos with the arm inpainted as a human hand) improved the translation model when added to real paired data, despite being imperfect.
The paper proposes a structured generalization scale: mild (new configurations of seen objects and skills), standard (new instances, unseen object-skill combinations) and strong (unseen object categories, unseen skills).

Main results

Table-top success rates (%) for HOPMan on mild / instance / unseen-combination / unseen-category / unseen-skill generalization: 68 / 58 / 50 / 40 / 25; goal-conditioned BC scored 43 / 20 / 13 / 0 / 0 and VRB 50 / 25 / 15 / 0 / 0.
In-the-wild success rates (%) for HOPMan: 60 / 40 / 40 / 30 / 25, versus 45 / 10 / 10 / 0 / 0 for BC and 50 / 15 / 10 / 0 / 0 for the MimicPlay-based MP baseline.
H2R, trained only on passive human video, scored 20 / 20 / 15 / 15 / 15 table-top and 15 / 10 / 15 / 15 / 10 in the wild; it was the only baseline above zero on strong generalization.
Translation model alone, given ground-truth hand-object masks (%): paired plus hallucinated data 85 / 65 / 60 / 60 / 25, paired only 85 / 60 / 50 / 30 / 15, hallucinated only 30 / 25 / 25 / 20 / 0.
Evaluation spans about 100 tasks built from 16 skills and 40 object types with 2 to 3 instances each.

Limitations

The authors state that the unseen tasks handled are still limited in complexity, and leave long-horizon tasks that compose multiple skills to future work.
They expect the framework to struggle with dexterous manipulation, since recovering precise hand and finger articulation from web video remains an open problem.
Success on strong generalization stays at 25 to 40% table-top and 25 to 30% in the wild.
Success is judged by manually inspecting how close the final object configuration is to the goal image, and the number of trials per setting is not stated.

Field impact

Finalist for the IEEE ICRA 2024 Best Paper Award in Robot Manipulation, with 82 citations. The paper links a project website with robot videos; it does not give a code repository.

Abstract

We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots. Unlike typical robot learning approaches which directly learn how a robot should act from interaction data, we adopt a factorized approach that can leverage large-scale human videos to learn how a human would accomplish a desired task (a human ‘plan’), followed by ‘translating’ this plan to the robot’s embodiment. Specifically, we learn a human ‘plan predictor’ that, given a current image of a scene and a goal image, predicts the future hand and object configurations. We combine this with a ‘translation’ module that learns a plan-conditioned robot manipulation policy, and allows following humans plans for generic manipulation tasks in a zero-shot manner with no deployment-time training. Importantly, while the plan predictor can leverage large-scale human videos for learning, the translation module only requires a small amount of in-domain data, and can generalize to tasks not seen during training. We show that our learned system can perform over 16 manipulation skills that generalize to 40 objects, encompassing 100 real-world tasks for table-top manipulation and diverse in-the-wild manipulation. https://homangab.github.io/hopman/

Open Source Robotics