Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation

Authors: William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, Phillip Isola · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

F3RM distills 2D CLIP or DINO features into a 3D NeRF feature field, so a robot can infer 6-DOF grasps and places from two demos per task and pick objects named in free text.

Why it matters

The method needs no fine-tuning of the vision models and no segmentation masks or object-pose assumptions, because it works over the whole feature field. The authors report that plugging a newer vision-language model into the pipeline takes a few minutes of engineering, with no retraining and no new demonstrations.

Problem

Pre-trained 2D image models carry rich semantic knowledge, but grasping and placing need accurate 3D geometry that 2D features lack. The paper asks how a robot can pick up objects it has never seen, given a few demonstrations or a text description, in cluttered scenes where objects differ from the demonstrated ones in shape, size, pose and material.

Proposed solution

The robot scans the table with an RGB camera on a selfie stick (50 images) and trains a NeRF that also renders dense features from DINO ViT or CLIP; CLIP patch features are extracted with the MaskCLIP reparameterization so they stay aligned with language. A 6-DOF gripper pose is encoded by sampling the occupancy-weighted feature field at 100 query points in the gripper frame, and poses are found by voxel pre-filtering plus Adam optimization against the averaged embedding of two demonstrations. For text commands, the CLIP text embedding retrieves the closest demonstrations and weights the pose cost toward regions that match the query.

Key insights

DINO and CLIP feature fields are complementary: DINO gives good part-level geometry but is confused by look-alike distractors and tends to overfit to color, while CLIP favors semantics over geometry: 21 of 27 CLIP failures versus 7 of 19 DINO failures came from grasp rotation or translation errors.
Dense CLIP features taken from the penultimate layer with MaskCLIP keep their alignment with the text encoder, so language queries work on the 3D feature field without fine-tuning CLIP.
Weighting features by NeRF occupancy (alpha) lets one query-point representation encode both local geometry and semantics, with no object segmentation or object poses.
Demonstrations on mugs and screwdrivers transferred to new categories such as bowls, measuring beakers, rolls of tape, whiteboard markers and utensils when the target was named in text.
Most language-guided failures (13 of 19) came from CLIP behaving like a bag-of-words and missing relations, attributes and ordinal information in the query.

Main results

Grasping and placing from two demonstrations per task, five tasks with ten test scenes each: CLIP ResNet features 39/50, CLIP ViT 34/50, DINO ViT 31/50.
Baselines on the same tasks: NeRF density 27/50, RGB 21/50, intermediate NeRF features 9/50, and MIRA 15/50.
Language-guided manipulation in 13 tabletop scenes: 31 of 50 queries succeeded; by category, color 7/10, material 7/10, relational 4/10, general 4/10, and out-of-distribution object categories 9/10.
Fitting the NeRF and feature field takes at most 90 s (80 s on average) on an NVIDIA RTX 3090, and each scene needs 60–120 MB, compared with 50–100s of GB per scene for surfel-based ConceptFusion.
Grasp pose optimization takes 15 s on average; in a view-count ablation, quality declines drastically below about 20 of the 50 scanned images.

Limitations

Stated by the authors: the automated scan and modeling pipeline takes a few minutes per scene and is the main bottleneck of the system.
The authors note that the 31/50 language-guided success rate is far from practical for industrial use.
CLIP's bag-of-words behavior caused failures on queries such as "black screwdriver" and "mug on a can of spam", and retrieving demonstrations by text sometimes needed prompt engineering.
Query-point distributions are tuned by hand for each task, and the best of five sampled query-point sets is chosen on development scenes.
Six of the 19 language-guided failures came from rotation or translation errors in the predicted grasp.

Field impact

Winner of the CoRL 2023 Best Paper Award. The official project page links the F3RM code repository.

Abstract

Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models. We present a few-shot learning method for 6-DOF grasping and placing that harnesses these strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Using features distilled from a vision-language model, CLIP, we present a way to designate novel objects for manipulation via free-text natural language, and demonstrate its ability to generalize to unseen expressions and novel categories of objects.

Open Source Robotics