Universal Visual Decomposer: Long-Horizon Manipulation Made Easy

Authors: Zichen Zhang, Yunshuang Li, Osbert Bastani, Abhishek Gupta, Dinesh Jayaraman, Yecheng Jason Ma, Luca Weihs · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

UVD finds subgoals in demonstration videos where pre-trained visual embedding distances stop decreasing, then uses them for goal-conditioned imitation and reward shaping.

Why it matters

UVD needs only video demonstrations and an off-the-shelf encoder, and the decomposition step adds under 0.1 s per trajectory on top of standard policy training. Subgoals found this way let a policy trained on some task orders carry out unseen orders of the same subtasks, which reduces the need to collect demonstrations for every task sequence.

Problem

Long-horizon manipulation suffers from compounding errors, large observation and action spaces, and weak learning signals, so tasks are usually split into subtasks. Existing decomposition methods need task-specific knowledge, such as proprioception or a known subtask structure, or require compute-heavy generative models trained on large datasets, so they cannot be applied out of the box to plain video demonstrations.

Proposed solution

UVD takes a frozen pre-trained visual encoder (such as VIP, R3M or LIV) and works backward from the final frame: within one subtask the embedding distance to the current goal should fall monotonically, so the latest frame where that monotonicity breaks becomes the previous subgoal, and the step repeats until the start of the video. The discovered subgoals relabel demonstrations for goal-conditioned behavior cloning and define an RL reward as the per-step decrease in embedding distance to the current subgoal. At test time, goals come from a nearest-neighbor lookup (single task) or from relaying a given subgoal sequence.

Key insights

Representations trained on short videos to encode task progress (VIP, R3M, LIV) give near-monotone embedding distances within one subtask; across several subtasks the distance curve plateaus at phase shifts, which marks subtask boundaries without labels.
VIP and LIV, trained explicitly for smooth temporal progress, gained the most from UVD, supporting the claim that temporally trained representations suit subgoal discovery better than static-image models such as CLIP and DINOv2.
Heuristic subgoal choices (a frame within a fixed window ahead, or 3 to 5 random frames) performed comparably in-domain but hurt out-of-domain generalization, because they give redundant, less meaningful subgoals.
Rewards defined against the distant final frame did not guide RL exploration on a four-stage task; per-subgoal rewards did.
Because embeddings are noisy, a minimum time interval between subgoals and smoothing of the distance curve (Nadaraya-Watson kernel regression) are needed in practice.

Main results

FrankaKitchen imitation with VIP and GCBC: out-of-domain success rose from 0.035 to 0.188 and out-of-domain completion from 0.236 to 0.566, with in-domain success unchanged (0.736 vs 0.737).
Out-of-domain success improved for every representation tested: R3M 0.014 to 0.084, LIV 0.008 to 0.066, CLIP 0.005 to 0.073, DINOv2 0.012 to 0.055, and with a GPT-style transformer policy on VIP 0.039 to 0.213.
FrankaKitchen RL with PPO: goal-conditioned RL using the final frame as goal reached 0.0 success; with UVD rewards, average/max success over 3 seeds was 0.65/1.0 with VIP and 0.649/1.0 with R3M.
Real Franka, out-of-domain success over 20 rollouts (GCBC vs GCBC + UVD): Apple-in-Oven 0.0 vs 0.25, Fries-and-Rack 0.0 vs 0.25, Fold-Cloth 0.0 vs 0.15.
With human interference during execution (10 rollouts), success went from 0.0 to 0.5 on Apple-in-Oven and from 0.2 to 0.9 on Fries-and-Rack.
Decomposition time on an RTX A6000: 0.0023 s for an average 227-frame FrankaKitchen episode and 0.011 s for a 698-frame 720p video, after encoding (0.155 s and 0.450 s).

Limitations

Validated only on fully observable manipulation tasks; the authors note that navigation and other partially observed embodied tasks may not yield intuitive or explainable subgoals.
Real-world out-of-domain tests are shorter than the training tasks: because the real subtasks are sequentially dependent, they test skipping a step a person has already completed.
Subgoal detection relies on a preset minimum interval between subgoals and a smoothing step, and goal relaying at inference relies on a distance threshold.
Real-robot evaluation covers three tasks, each with about 100 teleoperated demonstrations and 20 rollouts per setting.
Gains depend on the representation: in-domain success with CLIP and DINOv2 stayed between 0.32 and 0.39.

Field impact

Finalist for the IEEE ICRA 2024 Best Paper Award in Robot Vision, with 44 citations. The paper links a project page and gives Python pseudocode of the decomposition in its appendix; it does not give a code repository link.

Abstract

Real-world robotic tasks stretch over extended horizons and encompass multiple stages. Learning long-horizon manipulation tasks, however, is a long-standing challenge, and demands decomposing the overarching task into several manageable subtasks to facilitate policy learning and generalization to unseen tasks. Prior task decomposition methods require task-specific knowledge, are computationally intensive, and cannot readily be applied to new tasks. To address these shortcomings, we propose Universal Visual Decomposer (UVD), an off-the-shelf task decomposition method for visual long-horizon manipulation using pre-trained visual representations designed for robotic control. At a high level, UVD discovers subgoals by detecting phase shifts in the embedding space of the pre-trained representation. Operating purely on visual demonstrations without auxiliary information, UVD can effectively extract visual subgoals embedded in the videos, while incurring zero additional training cost on top of standard visuomotor policy training. Goal-conditioned policies learned with UVD-discovered subgoals exhibit significantly improved compositional generalization at test time to unseen tasks. Furthermore, UVD-discovered subgoals can be used to construct goal-based reward shaping that jump-starts temporally extended exploration for reinforcement learning. We extensively evaluate UVD on both simulation and real-world tasks, and in all cases, UVD substantially outperforms baselines across imitation and reinforcement learning settings on in-domain and out-of-domain task sequences alike, validating the clear advantage of automated visual task decomposition within the simple, compact UVD framework.

Open Source Robotics