Deep Drone Racing: Learning Agile Flight in Dynamic Environments

Authors: Elia Kaufmann, Antonio Loquercio, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, Davide Scaramuzza · CoRL 2018

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A CNN maps one onboard camera image to a goal direction and speed; a minimum-jerk planner and classic controller fly the quadrotor, so it races tracks with moving gates without a map or global pose.

Why it matters

Stability stays with a classic controller while the network only handles perception, so the learned part can run at about 10 Hz on a small onboard computer. Because commands are given in the body frame, the system does not suffer from the drift that affects visual-inertial odometry baselines at speed.

Problem

A racing drone that tracks a precomputed global trajectory needs highly accurate state estimation, and SLAM-style localisation drifts under motion blur and high acceleration. Such methods also assume a static world, so waypoints cannot be fixed in advance when gates move.

Proposed solution

A ResNet-based CNN takes a 300 × 200 RGB image and outputs a goal direction in normalised image coordinates plus a normalised speed. The goal is back-projected to a 3D point, a minimum-jerk state-interception trajectory is computed to reach it, and an existing low-level controller tracks it, recomputed in receding-horizon fashion. The network is trained by imitating an expert that follows a minimum-snap global trajectory, using data from several gate layouts so it copes with gates that move at test time.

Key insights

Predicting a waypoint and speed, rather than body rates, lets a classic controller keep the quadrotor stable; an end-to-end body-rate regressor could not pass more than one gate in simulation
Training on several static gate layouts was enough for the policy to fly through gates that move during flight
The learned policy is not bounded by its demonstrations: it copes with moving gates, which the expert it imitates cannot
Real-world training data can be collected safely by carrying the quadrotor through the track by hand while the expert labels each image
A single trained network covers a range of aggressiveness, since the user scales the predicted speed by a chosen maximum speed

Main results

Simulation, 8-gate 116 m track: the method reliably completes the task up to a maximum speed of 9 m/s, degrading gracefully at higher speeds; the VIO baseline fails from accumulated drift. Section 4.2 states five runs per setting, while the Figure 4 caption states 10 runs per speed
Simulation with moving gates at 8 m/s: no crashes with gate movement of up to 1.5 times the gate diameter, while the VIO baseline fails even for small gate motions
Real narrow-gap test: the network passed the gate in 100%, 80% and 20% of runs from initial angles of 0–30°, 30–70° and 70–90° (gate invisible at start in the last range), against 70%, 0% and 0% for a handcrafted window detector
Real occlusion test: the approach was robust to occlusions of up to 50% of the gate area, while the handcrafted baseline broke down at moderate occlusion
Real 4-gate 21 m track: higher success rate and speed than a VIO-based planning-and-tracking baseline; professional and intermediate human pilots were faster but crashed more often
Simulation comparison: the end-to-end body-rate baseline never passed more than one gate on a 4-gate 43 m track, even with 35k training samples versus 5k for the proposed method

Limitations

Performance will degrade if the environment's appearance changes substantially beyond what was seen in training
Training the perception system needs a significant amount of data collected in the application environment
Where trajectory optimisation cannot provide a policy to imitate, for example extremely tight turns, the learner is also likely to fail
Above 9 m/s in simulation, the limited prediction rate (30 Hz) cannot keep up with the large roll and pitch rates
Human pilots completed the real track faster, with more agile manoeuvres in sharp turns that the system cannot yet reason about

Field impact

Winner of the CoRL 2018 Best Systems Paper Award, with 166 citations. The paper links a supplementary video; it does not release code or data.

Abstract

Autonomous agile flight brings up fundamental challenges in robotics, such as coping with unreliable state estimation, reacting optimally to dynamically changing environments, and coupling perception and action in real time under severe resource constraints. In this paper, we consider these challenges in the context of autonomous, vision-based drone racing in dynamic environments. Our approach combines a convolutional neural network (CNN) with a state-of-the-art path-planning and control system. The CNN directly maps raw images into a robust representation in the form of a waypoint and desired speed. This information is then used by the planner to generate a short, minimum-jerk trajectory segment and corresponding motor commands to reach the desired goal. We demonstrate our method in autonomous agile flight scenarios, in which a vision-based quadrotor traverses drone-racing tracks with possibly moving gates. Our method does not require any explicit map of the environment and runs fully onboard. We extensively test the precision and robustness of the approach in simulation and in the physical world. We also evaluate our method against state-of-the-art navigation approaches and professional human drone pilots.

Open Source Robotics