Online Object Model Reconstruction and Reuse for Lifelong Improvement of Robot Manipulation

Authors: Shiyang Lu, Rui Wang, Yinglong Miao, Chaitanya Mitash, Kostas Bekris · ICRA 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Recognise objects the robot has handled before and reuse their partial TSDF models: success rises from 85% to 98% and fewer handoffs and re-scans are needed.

Why it matters

A robot that sees the same items repeatedly, as in warehouses or kitchens, can use earlier encounters to shrink shape uncertainty instead of starting from scratch. The paper shows this lowers the number of extra manipulation steps needed to place objects in constrained regions.

Problem

Pick-and-place systems for objects without known 3D models usually treat every object as new, even when the same object has been handled before. For placement in tight spaces they must assume a conservative volume for unseen parts, which forces extra handoffs and active-perception moves and sometimes leaves no safe grasp at all.

Proposed solution

Each manipulated object is stored as a 1 mm truncated signed distance function (TSDF) volume plus 64 clusters of ResNet50 colour features from different viewpoints. A new object is matched to the dataset by cosine distance, then a GPU particle filter registers the stored partial model and rejects poses that lie in observed free space or below the table. The same particle filter tracks the object in hand while new depth views are fused into its model.

Key insights

Occupancy-grid models were not precise enough to reuse across tasks; a 1 mm TSDF volume was
Pruning pose hypotheses that put the model in observed free space or under the support plane rejected the one false recognition before it caused a failed task
Representing an object by several viewpoint feature clusters instead of one mean feature raises recall from 0.211 to 0.886 at the chosen threshold
False negatives only cost efficiency, since the object is treated as novel, while false positives can fail a task, so the threshold is set for high precision
Rendering only a region of interest halves particle-filter iteration time compared with the prior full-image method

Main results

Task success 54/55 (98%) versus 47/55 (85%) for the baseline that treats every object as novel
Handoff actions in 20/55 tasks (36%) versus 37/55 (67%) for the baseline
Active perception actions in 8/55 tasks (14%) versus 35/55 (63%) for the baseline
Registering a stored model cut the conservative shape-uncertainty volume by 32% on average and by up to 75%
Object correctly recognised in 49/55 experiments, with 5 false negatives and 1 false positive that pose estimation rejected
Reconstructed models within 5 mm Chamfer distance of ground truth for all four objects; pose-estimation iterations take about 30 ms versus about 60 ms for the prior method with 625 particles

Limitations

The authors state the TSDF representation is not suited to thin structures, so objects such as bowls are hard to reconstruct and track
Cluttered scenes are left to future work; each task contains a single singulated object
Reconstructed models contain noisy interior points from tracking errors when the object is highly occluded
Evaluated on four objects in 55 tasks per system, and false negatives occur when the visible part of an object has not been seen before

Field impact

Finalist for the ICRA 2022 Outstanding Manipulation Paper award, with 17 citations. The authors state that the particle-filter pose estimator is open-sourced on GitHub as a 6DoF pose annotation tool.

Abstract

This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and performs online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning features of objects it has interacted with and updates their reconstructed models. Whenever an instance of a previously manipulated object reappears, the system aims to first recognize it and then register its previously reconstructed model given the current observation. This step greatly reduces object shape uncertainty allowing the system to even reason for parts of the objects that are currently not observable. This also results in better manipulation efficiency as it reduces the need for active perception of the target object during manipulation. To get a reusable reconstructed model, the proposed pipeline adopts: i) TSDF for object representation, and ii) a variant of the standard particle filter algorithm for pose estimation and tracking of the partial object model. Furthermore, an effective way to construct and maintain a dataset of manipulated objects is presented. A sequence of real-world manipulation experiments is performed to show how future manipulation tasks become more effective and efficient by reusing reconstructed models of previously manipulated objects that were generated on the fly instead of treating objects as novel every time.

Open Source Robotics