GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation

Authors: Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang, Minzhao Zhu · arXiv 2024 (technical report)

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A GPT-style model pre-trained to predict video on 38M web clips, then fine-tuned to output future frames and action trajectories; 97.7% success on 105 real tabletop tasks.

Why it matters

The report shows that web video can carry much of the knowledge a manipulation policy would otherwise need from robot data: GR-2 reaches 73.9% success on 105 tasks with only about 50 demonstrations per task and roughly doubles GR-1's success on unseen backgrounds and kitchens. Its bin-picking tests on 122 objects, including transparent, deformable and reflective items, target an industrial task that the authors note can be hard for model-based methods.

Problem

Collecting large-scale robot data is slow and real-robot systems scale poorly, which limits how general a manipulation policy can become. A generalist agent has to learn many skills and still cope with new backgrounds, environments, objects and instructions. The authors' earlier GR-1 showed that video generation pre-training helps policy learning, but it used 0.8 million videos and generalised weakly to unseen scenes.

Proposed solution

GR-2 is a language-conditioned GPT-style transformer: a frozen text encoder tokenises the instruction, a frozen VQGAN turns each frame into discrete tokens, and robot states (end-effector pose and gripper state) pass through linear layers. It is first pre-trained to predict future frames on 38 million text-video clips (over 50 billion tokens per the abstract, approximately 50 billion per Section 2.1) from human-activity datasets and public robot datasets, then fine-tuned on multi-view robot data to predict future images together with an action trajectory from a conditional VAE. On the real arm, a whole-body control algorithm smooths the Cartesian trajectory and converts it to joint commands executed at 200 Hz with collision constraints.

Key insights

Scaling pre-training from GR-1's 0.8 million videos to 38 million clips, together with an architecture built to carry pre-trained knowledge into fine-tuning, roughly doubled GR-1's success in the Unseen Backgrounds and Unseen Environments settings.
Generating action trajectories rather than single-step actions was found to be crucial for both trajectory smoothness and real-time performance.
Predicted videos align closely with real rollouts, so the action output effectively 'replays' the predicted video; the authors suggest improving video generation as a route to better actions.
Synthetic augmentation (diffusion-based object insertion, SAM-based background replacement, video regeneration) raised Unseen Environments success from 71.7% to 87.0%.
Video-prediction validation loss and real-robot success rate both improved as trainable parameters grew from 30M to 719M.

Main results

Multi-task learning: 97.7% average success on 105 tabletop tasks covering 8 skills in the Simple setting, after fine-tuning on about 40,000 teleoperated trajectories.
Generalisation: 71.4% on Unseen Backgrounds and 71.7% on Unseen Environments, about double GR-1; with data augmentation, 87.0% on Unseen Environments and 74.7% averaged over the three generalisation settings; 55.8% on Unseen Manipulation.
Low-data training with about 50 trajectories per task (about 1/8 of the data): 73.9% in the Simple setting, beating GR-1 in all three generalisation settings.
End-to-end bin picking trained on about 94,000 trajectories of 55 objects and tested on 122 objects (67 unseen): average success 79.0% versus 33.3% for GR-1.
CALVIN ABCD→D: 98.6% success for 1 task and 85.9% for 5 tasks in a row (GR-1: 94.9% and 73.1%), with average length rising from 4.21 to 4.64.
The default model has 230M parameters, 95M of them trainable; scaling runs use 30M, 95M, 312M and 719M trainable parameters.

Limitations

The authors identify unseen manipulation as the weakest area (55.8%); typical failures are not picking unseen objects of novel shapes and picking the wrong object when asked for an unseen one.
Their stated future work is improving generalisation and robustness of action prediction, especially for novel objects and new skills.
All real-robot results come from a single platform, a Kinova Gen3 arm with a Robotiq 2F-85 gripper; transfer to other embodiments is not tested.
The report gives no training compute, inference hardware or policy latency figures, and most per-setting numbers appear only in figures.
The text does not state any release of code, model weights or the robot datasets.

Field impact

This ByteDance Research technical report has 377 citations. It links a project page (gr2-manipulation.github.io); the text does not state a release of code, model weights or data.

Abstract

We present GR-2, a state-of-the-art generalist robot agent for versatile and generalizable robot manipulation. GR-2 is first pre-trained on a vast number of Internet videos to capture the dynamics of the world. This large-scale pre-training, involving 38 million video clips and over 50 billion tokens, equips GR-2 with the ability to generalize across a wide range of robotic tasks and environments during subsequent policy learning. Following this, GR-2 is fine-tuned for both video generation and action prediction using robot trajectories. It exhibits impressive multi-task learning capabilities, achieving an average success rate of 97.7% across more than 100 tasks. Moreover, GR-2 demonstrates exceptional generalization to new, previously unseen scenarios, including novel backgrounds, environments, objects, and tasks. Notably, GR-2 scales effectively with model size, underscoring its potential for continued growth and application. Project page: https://gr2-manipulation.github.io.

Open Source Robotics