Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles

Authors: Ionel Gog, Sukrit Kalra, Peter Schafhalter, Matthew A. Wright, Joseph E. Gonzalez, Ion Stoica · ICRA 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Modular, deterministic AV pipeline on a dataflow runtime: swap detectors, trackers, predictors or planners and see how their runtime affects end-to-end driving in CARLA or a real car.

Why it matters

Teams building driving or mobile-robot stacks can test a new detector or planner inside a complete pipeline and measure collisions and ride comfort rather than only offline accuracy. The paper shows that the best component choice changes with driving speed and scenario, which argues for evaluating components in context.

Problem

Autonomous-vehicle components are usually judged with offline metrics such as average precision that ignore runtime, even though a slow but accurate model can make the vehicle less safe. Existing open stacks such as Autoware and Apollo offer limited simulator interfaces without photo-realistic cameras, and their publisher-subscriber design makes deterministic replay of driving scenarios hard.

Proposed solution

Pylot structures the whole pipeline (detection, tracking, prediction, planning, control) as a dataflow graph of operators exchanging timestamped messages with watermarks, running on the authors' ERDOS system, which makes execution deterministic and lets components with different runtimes be swapped without retuning. It ships several reference implementations per component plus ground-truth 'perfect' versions from CARLA, and a pseudo-asynchronous mode that runs CARLA synchronously at over 200 Hz while delaying each control command by the pipeline's measured runtime. A ROS shim layer lets the same pipeline drive a real vehicle.

Key insights

Timely accuracy, which scores a module's output against the world at the moment the output is ready rather than when the input was captured, exposes how runtime erodes accuracy in a moving scene.
The best detector or tracker is context-dependent: fast, lower-accuracy detectors beat slow, high-accuracy ones at high speeds, and SORT, DeepSORT and DaSiamRPN each suit different situations.
In an emergency swerve scenario, planner configurations with lower runtime avoided collisions more often, while the Frenet Optimal Trajectory planner gave a smoother ride than RRT* at the cost of fewer avoided collisions.
A dataflow model with watermarks reduces the pipeline to a Kahn Process Network, which gives deterministic replay for debugging that publisher-subscriber systems lack.

Main results

A runtime of 10 ms cuts the mean timely mIoU of a perfect semantic segmentation component to about 0.75, below the mIoU of the top three Cityscapes submissions.
Mean timely AP50 of a perfect pedestrian detector halves with a 35 ms runtime.
With a 20 ms runtime, median timely mIoU is 28% lower at 40 m/s than at 10 m/s, and median timely AP50 drops from 1.0 at 10 m/s to 0 at 40 m/s.
In the occluded-pedestrian scenario (Table I), the slow FOT (P99 550 ms), RRT (P99 76 ms) and Hybrid A (P99 760 ms) configurations failed at every target speed from 16 to 22 m/s, while the fast configuration of each planner avoided the collision at 16 m/s.
Porting the pipeline from CARLA to a Lincoln MKZ test vehicle took 434 lines of Python code; Pylot itself is about 28,000 lines of Python.
The underlying dataflow system is reported to beat ROS by 30% in communication latency, and Pylot is described as the top submission on the CARLA Autonomous Driving Challenge HD map track.

Limitations

Reinforcement-learning approaches to driving are not yet supported; an OpenAI Gym interface is named as future work.
The end-to-end planner comparison uses ground-truth perception rather than the perception models, and only one emergency scenario.
Timely-accuracy studies use perfect perception components with emulated runtimes, not the runtimes of real models running in the loop.
The real-vehicle deployment is described as a port with off-the-shelf models, with no reported driving metrics on the real car.

Field impact

The paper was a finalist for the IEEE ICRA 2021 Best Paper Award on Unmanned Aerial Vehicles and has 107 citations. Pylot is released as open-source code, with reference and ground-truth component implementations and a suite of NHTSA-based test scenarios for CARLA.

Abstract

We present Pylot, a platform for autonomous vehicle (AV) research and development, built with the goal to allow researchers to study the effects of the latency and accuracy of their models and algorithms on the end-to-end driving behavior of an AV. This is achieved through a modular structure enabled by our high-performance dataflow system that represents AV software pipeline components (object detectors, motion planners, etc.) as a dataflow graph of operators which communicate on data streams using timestamped messages. Pylot readily interfaces with popular AV simulators like CARLA, and is easily deployable to real-world vehicles with minimal code changes. To reduce the burden of developing an entire pipeline for evaluating a single component, Pylot provides several state-of-the-art reference implementations for the various components of an AV pipeline. Using these reference implementations, a Pylot-based AV pipeline is able to drive a real vehicle, and attains a high score on the CARLA Autonomous Driving Challenge. We also present several case studies enabled by Pylot, including evidence of a need for context-dependent components, and per-component time allocation. Pylot is open source, with the code available at https://github.com/erdos-project/pylot.

Open Source Robotics