Interactive Object Segmentation in 3D Point Clouds

Authors: Theodora Kontogianni, Ekin Celikkan, Siyu Tang, Konrad Schindler · ICRA 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Users click positive and negative points directly in a 3D point cloud; a sparse-voxel network trained only on ScanNet refines the object mask and transfers to other indoor and outdoor datasets.

Why it matters

Robot perception models need labeled point clouds, and this method works on the point cloud alone, with no aligned images, and can be added to any point- or voxel-based 3D segmentation network. It also segments classes absent from its training set, and the authors show it can refine automatic instance masks and existing ground truth with a few clicks.

Problem

Dense 3D labels are expensive: scenes contain millions of points and 3D labeling interfaces are slow. Earlier interactive 3D methods took user feedback in 2D images, which needs aligned images with known camera poses and switching between 2D and 3D views, while fully supervised instance segmentation needs large training sets, does not generalise to unseen classes and gives masks too coarse to serve as ground truth.

Proposed solution

The user clicks points in the 3D scene; each click marks all points inside a small cube (5 cm edge, about 30 points on ScanNet) in one of two extra input channels, for positive and negative clicks. A MinkowskiEngine sparse-convolution network, initialised from ScanNetV2 semantic-segmentation weights, outputs a binary foreground/background mask and is fine-tuned with randomly simulated clicks; at test time a simulated user clicks the centre of the largest error region. Optional online adaptation updates the model from the user's clicks during labeling.

Key insights

Encoding clicks as small cubes fits existing 3D labeling tools that use bounding boxes; a 5 cm edge worked best (NOC@90 of 14.72, against 16.09 with 10 cm).
A model trained only on ScanNetV2 needed fewer clicks on S3DIS than on its own validation set, while SemanticKITTI (sparse, no colour, different sensor) needed many more.
Online adaptation from the user's clicks cut about 4 clicks on SemanticKITTI and helped S3DIS classes absent from ScanNet, such as beam and board.
Simulated clicks were representative of real annotators in a three-person user study.
Masks at 80% IoU still contain significant erroneous regions, so the authors argue ground-truth labeling should target about 90% IoU.

Main results

ScanNetV2 validation, clicks needed to reach 80 / 85 / 90% IoU: 8.3 / 10.6 / 13.6 on seen classes and 11.7 / 14.1 / 16.5 on unseen classes.
3D instance segmentation AP on ScanNetV2 benchmark classes: 61.8 with 5 clicks per object in Table 1 (the text gives 61.5) and 75.5 with 10 clicks, against 55.1 for class-agnostic HAIS and 49.4 for SSTNet.
Unseen ScanNetV2 classes: AP of 13.5 with 1 click, 33.1 with 3 clicks and 60.3 with 10 clicks, against 13.8 for class-agnostic HAIS.
S3DIS Area 5 without fine-tuning: 6.8 / 8.9 / 11.8 clicks to 80 / 85 / 90% IoU, and 6.5 / 8.6 / 11.5 with online adaptation.
SemanticKITTI: 19.4 / 19.5 / 19.6 clicks to 80 / 85 / 90% IoU, reduced to 15.1 / 15.4 / 15.9 with online adaptation.
User study: on the hard 'picture' class, IoU after 5 clicks was 49.6 for humans vs 32.0 for the simulator; on random objects, IoU after 10 clicks was 82.0 vs 84.8.

Limitations

The work is 3D perception tooling: it segments objects in point clouds with a person's clicks, and no robot is used in its experiments.
The authors list as future work enhancing the input with pre-trained semantic and instance backbones.
Masks from previous interactions are not yet fed back as input for the next correction.
Training samples clicks at random rather than with an iterative strategy that emulates a real user, and no loss handles the foreground/background class imbalance.
Sparse outdoor scans without colour need many more clicks: about 19.5 to reach 80–90% IoU on SemanticKITTI without adaptation.

Field impact

Finalist for the ICRA 2023 Outstanding Physical Human-Robot Interaction Paper award, with 36 citations. The licensed copy does not link code or data; it describes an annotation tool built for the user study but does not say it is released.

Abstract

Deep learning depends on large amounts of labeled training data. Manual labeling is expensive and represents a bottleneck, especially for tasks such as segmentation, where labels must be assigned down to the level of individual points. That challenge is even more daunting for 3D data: 3D point clouds contain millions of points per scene, and their accurate annotation is markedly more time-consuming. The situation is further aggravated by the added complexity of user interfaces for 3D point clouds, which slows down annotation even more. For the case of 2D image segmentation, interactive techniques have become common, where user feedback in the form of a few clicks guides a segmentation algorithm – nowadays usually a neural network – to achieve an accurate labeling with minimal effort. Surprisingly, interactive segmentation of 3D scenes has not been explored much. Previous work has attempted to obtain accurate 3D segmentation masks using human feedback from the 2D domain, which is only possible if correctly aligned images are available together with the 3D point cloud, and it involves switching between the 2D and 3D domains. Here, we present an interactive 3D object segmentation method in which the user interacts directly with the 3D point cloud. Importantly, our model does not require training data from the target domain: when trained on ScanNet, it performs well on several other datasets with different data characteristics as well as different object classes. Moreover, our method is orthogonal to supervised (instance) segmentation methods and can be combined with them to refine automatic segmentations with minimal human effort.

Open Source Robotics