Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping

Authors: Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, Ken Goldberg · CoRL 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Builds a LERF of the scene, masks the named object with DINO features, queries the part name only inside that mask, and re-ranks GraspNet grasps so the robot grasps the named part.

Why it matters

The method needs no part labels, affordance datasets, demonstrations or fine-tuning, and it handles long-tail queries such as 'ethernet dongle', 'cork' or 'martini glass' because it uses CLIP zero-shot. The part query is plain text, so a language model can supply the object and part from a task description such as 'cut the steak'.

Problem

Many objects must be grasped by a specific part, such as a knife by its handle or sunglasses by the frame, but learning-based grasp planners mostly consider geometry and judge success by whether the object was lifted. Earlier task-oriented grasping methods collect object-part or affordance datasets, which struggle to scale to many objects. LERF on its own highlights only the most identifying region of an object, and CLIP behaves like a bag of words, so 'mug' and 'mug handle' activate almost the same regions.

Proposed solution

A wrist camera captures about 60 images on a hemispherical path, and a Language Embedded Radiance Field (CLIP and DINO features distilled into a NeRF, built on Nerfstudio's Nerfacto) is trained in about 78 seconds. The object query localizes the object, a 3D object mask is grown by flood-filling DINO feature similarity, and the part query is evaluated only on points inside that mask (a conditional LERF query). GraspNet grasps generated from a hemisphere of virtual cameras are ranked by 0.95 × semantic score (median part relevancy inside the grasp volume) + 0.05 × GraspNet confidence.

Key insights

A 3D object mask from DINO features supplies the object grouping that LERF's CLIP relevancy lacks, so part queries cover the whole object, including oblong objects.
Restricting the part query to the object mask (conditional querying) counters CLIP's bag-of-words behaviour.
The semantic score must dominate the ranking (weight 0.95): the highest-confidence geometric grasp on the object mask lands on the correct part only 18% of the time.
Multi-scale CLIP features matter for parts: ConceptFusion, with one CLIP embedding per point, picks the right object about as often but the right part far less often.
Main failure modes are visually similar parts (teapot spout vs handle), small geometry (teacup handle, spray trigger), close categories (steak vs bread knife) and prompt wording ('bottle neck' localizes better than 'neck').

Main results

Part-oriented grasping with human-written queries (49 prompts, 12 scenes, Table 1): correct object 96%, correct part 82%, successful lift by the correct part 69%. The abstract and conclusion give the correct-part rate as 81%.
The number of physical objects is given as 31 in the abstract and Section 5, and as 39 in the introduction and the task-oriented results.
ConceptFusion with NeRF depth: 77% correct object and 39% correct part (no physical trials); single-view baselines on 20 prompts and 5 scenes: Semantic Abstraction 80% object / 35% part, OWL-ViT 85% object / 50% part.
With ChatGPT choosing object and part from a task description (majority vote over seven answers), Table 3 reports 96% correct object and 71% correct part; the Results text gives 92% for choosing the correct primitive and states the 71% as correct-object success.
Timing: scene capture 30 s, LERF training to 2k steps 78 s, LERF-TOGO query 10 s, with preprocessing spread over three NVIDIA 4090 GPUs during capture.

Limitations

The authors state that the end-to-end process takes a few minutes, which can be impractical for time-sensitive applications.
Groups of connected foreground objects are merged by the DINO flood-fill mask instead of being separated (stated by the authors).
If the object query matches several objects, such as identical mugs or a generic 'mug', the system arbitrarily chooses one (stated by the authors).
Referring or comparative expressions such as 'mug next to the plate' or 'biggest mug' are not handled (stated by the authors).
Each scene is reconstructed once and objects are then removed one by one without updating the representation; part selection is sensitive to wording, with LLM-generated part queries at 71% correct part against 82% for human queries.

Field impact

Finalist for the CoRL 2023 Best Paper / Best Student Paper Award and cited 159 times. The authors make code, data and the appendix available through lerftogo.github.io, which links a GitHub repository and a data folder.

Abstract

Grasping objects by a specific subpart is often crucial for safety and for executing downstream tasks. We propose LERF-TOGO, Language Embedded Radiance Fields for Task-Oriented Grasping of Objects, which uses vision-language models zero-shot to output a grasp distribution over an object given a natural language query. To accomplish this, we first construct a LERF of the scene, which distills CLIP embeddings into a multi-scale 3D language field queryable with text. However, LERF has no sense of object boundaries, so its relevancy outputs often return incomplete activations over an object which are insufficient for grasping. LERF-TOGO mitigates this lack of spatial grouping by extracting a 3D object mask via DINO features and then conditionally querying LERF on this mask to obtain a semantic distribution over the object to rank grasps from an off-the-shelf grasp planner. We evaluate LERF-TOGO's ability to grasp task-oriented object parts on 31 physical objects, and find it selects grasps on the correct part in 81% of trials and grasps successfully in 69%. Code, data, appendix, and details are available at: lerftogo.github.io

Open Source Robotics