OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication

Authors: Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, Jiaqi Ma · ICRA 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Open simulated V2V perception dataset (8 CARLA towns plus a digital Culver City), a benchmark of 4 LiDAR detectors × 4 fusion modes, and attentive feature fusion that holds AP at 4096× compression.

Why it matters

The dataset is reproducible from recorded configuration files and random seeds, so others can regenerate it or add sensors in identical driving situations. The compression experiments give a concrete bandwidth figure for feature sharing between vehicles: at 4096× compression a message takes about 5 ms over a 27 Mbps V2V link.

Problem

A single self-driving car misses objects that are heavily occluded or far away because its LiDAR returns on them are too sparse. Sharing sensing data between vehicles can fill these blind spots, but published vehicle-to-vehicle perception work used small, customised or unreleased datasets, so methods could not be benchmarked against each other.

Proposed solution

The authors generate OPV2V with the OpenCDA co-simulation framework on CARLA and SUMO: 73 scenes across 8 CARLA towns plus a digital town of Culver City, Los Angeles, built in RoadRunner to match data from a real vehicle, with 2-7 connected vehicles per frame, each carrying 4 cameras, a 64-channel LiDAR and GPS/IMU. They benchmark no, early, late and intermediate fusion with PIXOR, PointPillar, SECOND and VoxelNet, and propose an intermediate fusion pipeline that compresses shared features with an encoder-decoder and fuses them with self-attention over the features each vehicle holds for the same spatial location.

Key insights

Sharing data between vehicles raises AP at IoU 0.7 over single-vehicle detection for every detector and fusion strategy on both test sets; the paper reports gains of at least 10%.
Early fusion generally beats late fusion because it keeps more of the raw sensor measurements.
Attentive intermediate fusion gives the best AP on both test sets for all four detectors except PIXOR on Culver City, where early fusion is higher.
AP grows with the number of connected vehicles in a dense intersection, but the gain flattens after 4 vehicles because four viewpoints already cover most blind spots.
All models score lower on the Culver City digital town than on CARLA towns, which the authors attribute to more realistic, congested rush-hour traffic and the resulting domain gap.

Main results

Dataset: 73 scenes, 11,464 frames (about 40K counting all connected vehicles), 232,913 annotated 3D vehicle boxes, 249.4 GB, on average 2.89 connected vehicles per frame (minimum 2, maximum 7). The v1 abstract states 111,464 frames, which conflicts with 11,464 in the v1 body text, Table I and the 6764/1981/2719 split; the current arXiv abstract states 11,464.
VoxelNet with attentive intermediate fusion: AP 0.906 / 0.864 at IoU 0.5 / 0.7 on CARLA towns and 0.854 / 0.775 on Culver City, versus 0.688 / 0.526 and 0.605 / 0.431 without fusion.
Intermediate fusion with PointPillar: 0.908 / 0.815 (CARLA) and 0.854 / 0.735 (Culver City); with SECOND: 0.893 / 0.826 and 0.875 / 0.760.
Exception: PIXOR on Culver City reaches 0.734 / 0.558 with early fusion versus 0.716 / 0.549 with intermediate fusion.
With a 4096× compression rate, VoxelNet intermediate fusion loses about 3% AP at IoU 0.7 and still outperforms early and late fusion; delivery time is about 5 ms at 27 Mbps.
Training all 16 models took 14 days on 4 RTX 3090 GPUs; train/validation/test splits are 6764/1981/2719 frames.

Limitations

The dataset currently covers only the vehicle detection task; more tasks and sensor suites are left to future work.
The data is simulated; the real-world link is about 600 frames in 4 Culver City scenes reproduced in a digital town and used only for validation, and adding real-world data is planned, not done.
Models trained on CARLA towns score lower on Culver City, showing a domain gap to realistic traffic.
The benchmark uses LiDAR detectors only; multi-modal fusion and vehicle-to-infrastructure settings are left to future work.
The 5 ms delivery figure is computed from a 27 Mbps broadcast rate at 300 m taken from a cited V2V protocol overview, not measured.

Field impact

652 citations. The paper publishes the dataset generation code in the OpenCDA repository (data collection branch) and states that the dataset, benchmark methods and code will be released on the project page; the official project page links the data and the OpenCOOD code repository.

Abstract

Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle perception. It contains over 70 interesting scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes, collected from 8 towns in CARLA and a digital town of Culver City, Los Angeles. We then construct a comprehensive benchmark with a total of 16 implemented models to evaluate several information fusion strategies (i.e. early, late, and intermediate fusion) with state-of-the-art LiDAR detection algorithms. Moreover, we propose a new Attentive Intermediate Fusion pipeline to aggregate information from multiple connected vehicles. Our experiments show that the proposed pipeline can be easily integrated with existing 3D LiDAR detectors and achieve outstanding performance even with large compression rates. To encourage more researchers to investigate Vehicle-to-Vehicle perception, we will release the dataset, benchmark methods, and all related codes in https://mobility-lab.seas.ucla.edu/opv2v/.

Open Source Robotics