Instance Segmentation for Autonomous Log Grasping in Forestry Operations

Authors: Jean-Michel Fortin, Olivier Gamache, Vincent Grondin, François Pomerleau, Philippe Giguère · IROS 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A new masked dataset of wood logs shows that a pixel-level Transformer segments cluttered, crooked logs far better than box-proposal networks, a first perception step toward autonomous forwarders.

Why it matters

A crane cannot grasp a log it cannot localise, and per-log masks give a usable grasp point even for crooked logs. The results show which detector family suits cluttered, elongated objects, which also applies to other field and agricultural picking tasks.

Problem

Earlier work on automating log picking with forwarder cranes assumed log poses were known and gave little attention to perception. Logs are elongated, randomly oriented, overlapping and often crooked, so axis-aligned boxes take in neighbouring logs and a box centre can fall off a bent log; no public dataset of segmented logs existed.

Proposed solution

The authors collected images from dashcams in working forwarders near Lake Saint-Jean, Quebec, plus camera photos at lumberyards and frames from public videos, and annotated the top logs with polygon masks. They compared Mask R-CNN, Rotated Mask R-CNN with oriented anchors, and Mask2Former under 5-fold cross-validation, and studied how data size and input resolution affect Mask2Former.

Key insights

Axis-aligned region proposals enclose neighbouring logs, which degrades masks; oriented anchors help but Mask2Former, with no region proposals, does much better
Logs have a mean aspect ratio of 8.3 and up to 24, far from the 2:1 or 1:2 anchor ratios usually used
mAP grows with training set size following a power law, about 6.87% per doubling, so more data should help until a plateau
Using an earlier-trained network to pre-annotate images saved around 40% of labelling time per image
Masks are needed rather than oriented boxes because the centre of an oriented box can lie off a crooked log

Main results

Mask mAP: Mask R-CNN 19.03, Rotated Mask R-CNN 31.83, Mask2Former 56.05 with a Swin-B ImageNet-1k backbone and 57.53 with ImageNet-22k
Average recall 28.60, 36.95, 64.06 and 65.16, and F1 0.23, 0.34, 0.60 and 0.61 for the same four models
Inference speed of 12.30 fps for Mask R-CNN, 5.44 fps for Rotated Mask R-CNN, and 8.33 and 8.34 fps for the two Mask2Former variants, on an RTX 3090
At mask IoU 0.5, Mask2Former reaches precision above 80% while keeping 80% recall
Shrinking Mask2Former input to 256 × 256 raises speed to 24.05 fps; around 512 × 512 speed triples for about a 5% drop in precision
Logs per image are given as ranging from 1 to 30 in the Fig. 2 caption and from 1 to 29 in the text, with an average of 11.3

Limitations

The approach is single-frame, which the authors call a worst-case scenario; tracking logs across video frames is left to future work
The assumption that an IoU of 0.5 is enough for a successful grasp has not yet been verified experimentally
No grasping was performed: the paper evaluates perception only, on 220 images
Only the logs on top of a pile are annotated and detected

Field impact

Finalist for the IROS 2022 Best Paper Award on Agri-Robotics (sponsored by YANMAR), with 40 citations. The paper states that the TimberSeg 1.0 dataset will be made public upon publication.

Abstract

Wood logs picking is a challenging task to automate. Indeed, logs usually come in cluttered configurations, randomly orientated and overlapping. Recent work on log picking automation usually assume that the logs' pose is known, with little consideration given to the actual perception problem. In this paper, we squarely address the latter, using a data-driven approach. First, we introduce a novel dataset, named TimberSeg 1.0, that is densely annotated, i.e., that includes both bounding boxes and pixel-level mask annotations for logs. This dataset comprises 220 images with 2500 individually segmented logs. Using our dataset, we then compare three neural network architectures on the task of individual logs detection and segmentation; two region-based methods and one attention-based method. Unsurprisingly, our results show that axis-aligned proposals, failing to take into account the directional nature of logs, underperform with 19.03 mAP. A rotation-aware proposal method significantly improve results to 31.83 mAP. More interestingly, a Transformer-based approach, without any inductive bias on rotations, outperformed the two others, achieving a mAP of 57.53 on our dataset. Our use case demonstrates the limitations of region-based approaches for cluttered, elongated objects. It also highlights the potential of attention-based methods on this specific task, as they work directly at the pixel-level. These encouraging results indicate that such a perception system could be used to assist the operators on the short-term, or to fully automate log picking operations in the future.

Open Source Robotics