VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Authors: Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, Li Fei-Fei · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

An LLM writes Python that fills 3D affordance and avoidance voxel maps from vision-language detections; a planner turns them into closed-loop 6-DoF trajectories with no added training.

Why it matters

No component is trained for the tasks: knowledge from the language model is grounded directly in the robot's 3D observation space and consumed by a standard planner. Because the planner replans from fresh observations, execution tolerates disturbances such as moved targets or a drawer pulled open again after the robot closed it.

Problem

LLM-based robot planners usually call a library of pre-defined motion primitives, and acquiring those skills is a bottleneck because large-scale robot data is scarce. LLMs also cannot directly output high-frequency, high-dimensional control actions as text. The paper asks how to use LLM knowledge at the level of fine-grained actions, for open-set instructions and objects, without per-task data collection or hand-designed primitives.

Proposed solution

Given an instruction and RGB-D observations, GPT-4 generates Python code that calls perception APIs (OWL-ViT detection, Segment Anything masks, XMem tracking) and NumPy operations to fill 100×100×100 voxel maps for affordance, avoidance, end-effector rotation, velocity and gripper state. A greedy-search motion planner finds collision-free end-effector waypoints on a cost map built from the weighted affordance and avoidance maps, executes the first waypoint and replans at 5 Hz in a model predictive control loop. For contact-rich tasks, the zero-shot trajectories bias exploration while an MLP dynamics model is learned online.

Key insights

LLMs are poor at emitting raw control actions but good at inferring affordances and constraints, and they can express these as code that writes values into 3D voxel maps.
Composing spatial value maps under a joint optimization generalized better than an LLM that parameterizes a fixed list of primitives (a Code as Policies variant) and better than a U-Net trained to predict costmaps.
Zero-shot trajectories for contact-rich tasks are often meaningful but insufficient; used as an exploration prior they make learning a dynamics model practical.
In simulation VoxPoser had the lowest specification error of the compared methods; in the real world most errors came from perception, for example the detector's sensitivity to initial object poses and object parts.

Main results

Real world, five tasks with 10 trials each: 88.0% success in static scenes and 70.0% with disturbances, versus 24.0% and 0.0% for LLM + primitives.
Simulation (13 tasks, 2766 unique instructions, 20 episodes per task), object-interaction / spatial-composition success: 64.0% / 77.5% with seen instructions and attributes, 60.0% / 58.8% with unseen attributes, and 65.0% / 76.7% with unseen instructions and attributes.
In the unseen-instruction, unseen-attribute setting the baselines reached 0.0% / 0.0% (U-Net + motion planner) and 17.5% / 25.0% (LLM + primitives).
Contact-rich simulated tasks: zero-shot success of 6.7% (door), 3.3% (window) and 18.3% (fridge) rose to 88.3%, 80.0% and 91.7% after about 142 s, 137 s and 71 s of online interaction with the zero-shot prior; exploring without the prior exceeded the 12-hour limit.
The real setup replans at 5 Hz while two Azure Kinect cameras stream RGB-D at 20 Hz.

Limitations

Stated by the authors: VoxPoser relies on external perception modules, which limits tasks that need holistic visual reasoning or understanding of fine-grained object geometry.
Stated by the authors: a general-purpose dynamics model is still needed to handle contact-rich tasks with the same level of generalization; real object-centric tasks used a heuristic planar-pushing model and a pre-defined pushing primitive.
Stated by the authors: the motion planner considers only end-effector trajectories, while whole-arm planning is likely a better design.
Stated by the authors: LLM prompts are engineered by hand (5–20 example queries per language model program).
For most tasks the scene is assumed static apart from replanning, and decomposition of instructions into sub-tasks by the LLM is not studied in depth.

Field impact

Cited 735 times (count as given). The official project page links the VoxPoser code repository, and the paper publishes its full LLM prompts on the project website.

Abstract

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io

Open Source Robotics