Foundation models in robotics: Applications, challenges, and the future

Authors: Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, Mac Schwager · The International Journal of Robotics Research (IJRR) 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence.

TL;DR

Survey of foundation models in robotics, grouped into decision-making and control, perception and embodied AI, closing with eight open challenges from data scarcity to safety and real-time use.

Why it matters

Engineers deciding whether to use an LLM, VLM or robot transformer in a robot can use the survey's categories and its Table II, which lists parameter counts and inference rates such as 1-3 Hz for RT-2 against 20 Hz for Gato. The challenges section names practical deployment concerns: cloud-only models and network latency, hallucination, calibrated uncertainty and safety testing. The authors also state that many decision-making results are hard to compare because they depend on custom hardware stacks or non-physics simulators.

Problem

Deep learning models in robotics have usually been trained on small datasets gathered for one task, which limits how well they adapt to new tasks and settings. Foundation models pretrained on internet-scale data have generalised well in language and vision, and robotics researchers have started plugging them into, adapting them for, or building them for robots. Work in this area is spread across perception, planning and control, with no single map of what has been tried and what remains unsolved.

Proposed solution

This is a survey and proposes no method of its own. It limits its scope to background papers, robotics papers (foundation models used plug-and-play, fine-tuned, or newly built for robots) and robotics-adjacent papers, and covers foundation-model background, robot decision-making and control (language-conditioned imitation learning, language-assisted RL, value learning, LLM task planning and code generation, in-context learning, robot transformers, open-vocabulary navigation and manipulation), perception (open-vocabulary detection, segmentation, 3D scene representation, affordances, predictive models) and embodied AI with its simulators. It closes with challenges and future directions, and tabulates size, inference speed and training hardware for robot-specific pretrained models.

Key insights

The survey sorts robotics work into three ways of using foundation models: plug-and-play use of existing models, adapting or fine-tuning them, and building new robot-specific foundation models.
Robot physical skills learned by current robot transformers are limited to the distribution of skills present in the robot data; the authors suggest motion data from human videos as one route past this.
Many foundation models are reached only through cloud APIs, so latency depends on networks the robot does not control; the authors suggest local fall-back autonomy and distillation into smaller onboard models.
For safety-critical robots the authors argue that uncertainty estimates need to be calibrated in the frequentist sense, and present KnowNo's use of conformal prediction as a case study.
Non-physics simulators such as Gibson, Habitat, RoboTHOR and VirtualHome trade low-level physics for visual fidelity, which the authors link to a large sim-to-real gap and poor reproducibility.

Main results

As a survey it reports no experiments of its own; its reference list runs to 233 entries across background, robotics and robotics-adjacent papers.
Table II lists robot-specific pretrained models with size and inference rate, for example RT-1 at 35M parameters and 3 Hz, RT-2 and RT-2-X at 55B parameters and 1-3 Hz on a multi-TPU cloud service, Gato at 1.2B and 20 Hz, and ViNT at 31M and 4 Hz.
The same table lists PACT (12M parameters) at 10 Hz on an Nvidia Xavier NX edge computer and 50 Hz on 8 Nvidia V100 GPUs, and PaLM-E-562B with 1 Hz language subgoals plus 5 Hz low-level control.
The survey reports that RT-1 was trained on over 130k real-world robot episodes covering more than 700 tasks, collected over 17 months with 13 robots.
The survey reports that the RT-X dataset combines data from 22 robots and 21 institutions, covering 527 skills (160,266 tasks).
It lists eight challenge areas: data scarcity, real-time performance, limits of multimodal representation, uncertainty quantification, safety evaluation, plug-and-play versus building new models, high variability in robotic settings, and benchmarking and reproducibility.

Limitations

The authors call this version a preliminary release that they intend to update, so it captures the field as of December 2023.
Scope is limited to three categories of papers (background, robotics, robotics-adjacent), and the authors note a much narrower scope than broader foundation-model surveys.
The authors state that papers in the decision-making section are harder to assess and compare than those in the perception or embodied AI sections, because they rely on custom hardware stacks or non-physics simulators.
Table II has empty fields where the source papers report no data, so inference speed and hardware are not available for every model.

Field impact

The survey has 471 citations. Alongside it the authors released a GitHub project, Awesome-Robotics-Foundation-Models, which they describe as a preliminary release they intend to keep updating.

Abstract

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper (Preliminary release. We are committed to further enhancing and updating this work to ensure its quality and relevance) can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models

Open Source Robotics