Joint 3D Proposal Generation and Object Detection from View Aggregation

Authors: Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, Steven Waslander · IROS 2018

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A two-stage 3D detector that fuses LIDAR bird's-eye-view and image features in both the proposal network and the detector, and regresses an explicit heading vector to remove ±π orientation errors.

Why it matters

Fusing image features at the proposal stage raises recall on pedestrians and cyclists, the classes a LIDAR-only proposal stage misses. The network runs in 0.08 s per frame with 2 GB of GPU memory, which the authors present as suitable for use on a vehicle, and they run it on their own car with a different camera and LIDAR.

Problem

On a sparse LIDAR bird's-eye-view map, a pedestrian covers only about 8 × 6 pixels, so after downsampling a single feature-map pixel holds too little information to propose small objects. Earlier 3D detectors such as MV3D also infer heading from box corners, which leaves orientation ambiguous by ±π, and the KITTI 3D metrics do not penalise that error.

Proposed solution

Two VGG16-based feature extractors produce full-resolution feature maps from a six-channel BEV map and the RGB image. A region proposal network crops and resizes features for each 3D anchor from both views, after 1 × 1 convolutions cut memory, and fuses them by element-wise mean. A second-stage network regresses boxes in a 10-value four-corner-plus-two-heights encoding, plus an explicit (cos θ, sin θ) orientation vector used to pick the correct heading.

Key insights

Adding image features to the proposal network matters most for small classes: it raises pedestrian and cyclist recall but gives no gain for cars, which occupy enough of the BEV map
Feature crops for a single pixel are not enough for small objects: 1-pixel crops dropped AP by 26.9% for pedestrians and 32.6% for cyclists versus 9.5% for cars
Heading inferred from box corners alone is ambiguous by ±π; an explicit orientation vector resolves it
KITTI's AP metric does not penalise orientation flips that keep IoU above threshold, so the authors propose Average Heading Similarity (AHS)
1 × 1 convolutions before cropping cut per-view memory for anchor feature crops by 256×, so tens of thousands of anchors fit in a few megabytes

Main results

KITTI validation, car, moderate: 74.19% 3D AP and 73.97% AHS, against MV3D's 72.35% AP and 43.75% AHS (Table 1); the text gives the AHS gains over MV3D as 30.28% (moderate) and 27.54% (hard), and the introduction as 29.46% on the car class
KITTI test, car 3D AP (easy/moderate/hard): 71.41 / 62.87 / 56.61 versus MV3D's 71.09 / 62.35 / 55.12; pedestrian 29.45 / 23.42 / 22.22 and cyclist 35.94 / 29.04 / 26.44 (Table 2)
Proposal recall: 86% 3D recall on cars with 10 proposals and 91% with 50, where MV3D reported needing 300; maximum car recall for 3DOP and Mono3D is 73.87% and 65.74%
Fusion RPN versus BEV-only RPN: +6.9% AP for pedestrians and +9.4% for cyclists in the final detection (Table 3)
About 38.07 million parameters (about 37% of MV3D), 186.28 billion FLOPs per frame, 0.08 s per frame on a TITAN Xp (4.5× faster than MV3D), 2 GB GPU memory at inference
Ranking: the introduction says AVOD ranked third on all three KITTI classes at submission, while the results section says it currently ranks first on all three in 3D and BEV AP

Limitations

The KITTI test server does not compute AHS, so the orientation advantage is shown only on the validation split
No published method gave pedestrian and cyclist 3D results at the time, so 3D detection comparisons are for cars only
On the test set the AP gap to MV3D is small (0.52% and 1.49% on moderate and hard), and MV3D has higher BEV AP on easy cars (86.02 vs 85.51)
Pedestrian and cyclist 3D AP stays far below car AP (23.42 and 29.04 versus 62.87 on moderate test)
Deployment on the authors' vehicle is shown only qualitatively

Field impact

1739 citations. The paper links a video of qualitative results; it does not name a code or data release.

Abstract

We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produces state of the art results on the KITTI 3D object detection benchmark [10] while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles.

Open Source Robotics