Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

VIMA: General Robot Manipulation with Multimodal Prompts

Authors: Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, Linxi Fan · ICML 2023

Licences

  • Open Access
  • Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Casts manipulation tasks as interleaved text-and-image prompts, adds the VIMA-Bench benchmark, and proposes an object-token, cross-attention agent that generalises better than Gato-style designs.

Why it matters

One prompt interface covers goal reaching, one-shot video imitation, novel-concept grounding and constraint satisfaction, so a single policy can be trained and compared across them. The ablations give concrete design guidance: variable-length object tokens and cross-attention conditioning beat raw-pixel tokenisers and decoder-only conditioning, most of all for small models and novel tasks.

Problem

Robot tasks are specified in different forms, such as one-shot demonstrations, language instructions and visual goals, and these are usually tackled by specialised models with distinct architectures, objectives and data pipelines. No benchmark offered a suite of multimodal-prompt tasks with a systematic test of generalisation.

Proposed solution

Tasks are written as prompts that interleave text and images. VIMA encodes the prompt with a pre-trained T5-Base encoder, turns images into object tokens (Mask R-CNN bounding boxes plus ViT-encoded crops), and decodes discretised end-effector poses for pick-and-place or push with a transformer that alternates causal self-attention and cross-attention to the prompt. It is trained by behaviour cloning on VIMA-Bench, a Ravens/PyBullet extension with 17 task templates, 650K successful expert trajectories (50K per task, Section 3; the abstract rounds this to 600K+) and a four-level generalisation protocol.

Key insights

  • Object tokens outperform raw-pixel tokenisers (image patches, Image Perceiver, single-image token); the authors hypothesise pixel tokenisers spend capacity parsing objects from pixels.
  • Downsampling objects to a fixed number of tokens (Object Perceiver) is significantly worse than passing the variable-length object sequence, even with the same bounding-box inputs.
  • Cross-attention conditioning matters most at small model sizes and on L4 novel tasks; a decoder-only variant reaches comparable performance in larger models.
  • Prompt-encoder size (T5 small 30M, base 111M, large 368M) made no significant difference, so T5-Base is the default.
  • With object augmentation (randomly injected false-positive detections), the fine-tuned Mask R-CNN loses little performance compared with ground-truth boxes.

Main results

  • 200M-parameter controller: 81.5% (L1), 81.5% (L2), 78.7% (L3) and 48.6% (L4) average success on VIMA-Bench, against 58.0%, 53.1%, 46.0% and 16.8% for VIMA-Gato.
  • L4 (novel tasks) with a 20M controller: 48.8% for VIMA against 12.3% for VIMA-Gato, the basis of the 'up to 2.9×' relative improvement.
  • Trained on 10% of the data, VIMA reaches 46% on L4, against 12.2% for VIMA-Gato trained on the full dataset (the '2.7×' figure).
  • With 1% of the data, VIMA performs similarly to baselines trained with 10× more data on L1 and L2.
  • Moving to harder levels, VIMA shows minimal regression from L1 to L2 and L3 while baselines degrade by as much as 20%; its drop on L4 is half that of the baselines.

Limitations

  • Relies on a separate object detector and inherits its errors, which may arise under occlusion or with out-of-distribution object forms.
  • Limited simulator realism and task complexity: VIMA-Bench targets multi-task ability, generalisation and multimodal understanding, not physical realism.
  • Limited action primitives: only the high-level pick-and-place and push/wipe actions from Transporter; lower-level control such as joint torques would need modifications.
  • All experiments are in simulation; the paper reports no real-robot experiments of its own.
  • All methods, VIMA included, degrade significantly on L4 novel tasks, where VIMA stays below 50% average success.

Field impact

Published at ICML 2023. Semantic Scholar counts 655 citations (2026-11-14). The authors open-source the simulation environment (VIMA-Bench), the training dataset, the algorithm code and pre-trained model checkpoints.

Abstract

Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to 2.9× task success rate given the same training data. With 10× less training data, VIMA still performs 2.7× better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/

Links

  • arXiv 2210.03094: https://arxiv.org/abs/2210.03094
  • PDF: https://arxiv.org/pdf/2210.03094v2
  • Code: https://github.com/vimalabs/VIMA
  • Project page: https://vimalabs.github.io/

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.