Open-Vocabulary Affordance Detection in 3D Point Clouds

Authors: Toan Nguyen, Minh Nhat Vu, An Vuong, Dzung Nguyen, Thieu Vo, Ngan Le, Anh Nguyen · IROS 2023

Licences

Open Access
Paper: CC0 1.0 Universal

Everything below draws on the full paper, which its authors released under CC0 1.0 Universal; its figure is credited to them.

TL;DR

Matches per-point PointNet++ features to frozen CLIP text embeddings, so a robot can query a point cloud for any affordance word, including ones absent from training.

Why it matters

A robot can be told which part of an object to act on in plain words and get a 3D region it can pass straight to grasp detection, inverse kinematics and trajectory planning, without retraining for every new verb. Inference takes about 100 ms, and the authors run it on a KUKA LBR iiwa with a RealSense D435i and a Robotiq 2F-85 gripper.

Problem

Existing affordance detectors, on images or point clouds, are trained on a fixed set of affordance labels, so a robot cannot ask for a new one without new annotation. Image-based detectors also need an extra step to lift 2D results into 3D, and earlier one-shot or few-shot affordance methods still need annotated examples and work only on 2D images.

Proposed solution

OpenAD runs a PointNet++ backbone to produce a 512-D embedding for each of 2048 input points and encodes each affordance label with the frozen CLIP ViT-B/32 text encoder. Cosine similarity between every point embedding and every label embedding, scaled by a learnable temperature, gives a per-point softmax over whatever labels are supplied; training uses a class-weighted negative log-likelihood loss. At inference any text can be passed as a label.

Key insights

Aligning point features with a language model's text embeddings lets semantically related, unseen labels land on the right regions without a single annotated example
The choice of text encoder matters for open-vocabulary results: CLIP ViT-B/32 beat BERT on mIoU with both backbones, while closed-set results were about equal
Adding the language-driven head does not cost closed-set accuracy: OpenAD stays within a fraction of a point of dedicated point-cloud segmentation networks
Because each point takes the best label from the current label set, adding a more specific label (e.g. 'grasp' after 'contain') re-labels the relevant points

Main results

Zero-shot, full-shape (3D AffordanceNet re-labelled to 37 classes): mIoU 14.37 vs 9.97 for ZSLPC, 6.46 for 3DGenZ and 3.86 for TZSLPC; Acc 46.31 and mAcc 19.51, both the highest
Zero-shot, partial-view: mIoU 12.50 vs 9.52 for ZSLPC. The text states a 3.98% margin, while Table I's numbers give 2.98 points; Acc 45.25 and mAcc 17.37 are also the highest
Closed-set 3D AffordanceNet (Table II), full-shape: mIoU 42.00 vs 42.09 DGCNN, 41.26 PointNet++ and 41.26 Point Transformer; highest Acc at 69.03
Closed-set, partial-view: mIoU 41.87 vs 41.93 DGCNN, 41.10 PointNet++, 40.51 Point Transformer; highest Acc at 69.91
Ablation (open-vocabulary, full-shape): PointNet++ with CLIP 14.37 mIoU vs 10.55 with BERT; DGCNN with CLIP 10.88 vs 9.43 with BERT
Inference takes about 100 ms on average; real-robot demonstrations include holding and raising a bag, wrapping a bottle and pushing an earphone

Limitations

The authors state the method is still far from detecting completely unseen affordances and show failure cases on seen objects
The authors show false positives where OpenAD finds affordances the object does not offer (display on a bag, support on a microwave, openable on a hat)
Absolute zero-shot scores remain low (mIoU 14.37 full-shape, 12.50 partial-view), and the authors call for better vision-language correlation (e.g. cross-attention), stronger backbones and a larger affordance dataset
The real-robot experiments are demonstrations without reported success rates
Input is a single segmented object sampled to 2048 points; the robot pipeline relies on a separate object localisation step

Field impact

Named on the IROS 2023 combined finalist list for the Best Overall Paper and Best Student Paper awards, with 66 citations. The paper states the authors will release source code; no link is given in the licensed copy.

Abstract

Affordance detection is a challenging problem with a wide variety of robotic applications. Traditional affordance detection methods are limited to a predefined set of affordance labels, hence potentially restricting the adaptability of intelligent robots in complex and dynamic environments. In this paper, we present the Open-Vocabulary Affordance Detection (OpenAD) method, which is capable of detecting an unbounded number of affordances in 3D point clouds. By simultaneously learning the affordance text and the point feature, OpenAD successfully exploits the semantic relationships between affordances. Therefore, our proposed method enables zero-shot detection and can detect previously unseen affordances without a single annotation example. Intensive experimental results show that OpenAD works effectively on a wide range of affordance detection setups and outperforms other baselines by a large margin. Additionally, we demonstrate the practicality of the proposed OpenAD in real-world robotic applications with a fast inference speed (≈100 ms).

Open Source Robotics