TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models

Authors: Samuel Triest, Matthew Sivaprakasam, Sean J. Wang, Wenshan Wang, Aaron M. Johnson, Sebastian Scherer · ICRA 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A 5-hour, 630-trajectory off-road dataset from a modified Yamaha Viking ATV with seven modalities, plus latent dynamics-model benchmarks; exteroceptive inputs help most on rough terrain.

Why it matters

Learning dynamics models needs on the order of hundreds of thousands of interactions, and collecting them off-road is risky for the vehicle and its occupants. The dataset pairs actions with proprioceptive and exteroceptive data, so models can be trained and compared on real vehicle–terrain interaction instead of simulation. The benchmarks also show where current architectures fall short, such as auxiliary observation prediction adding little under real-world partial observability.

Problem

Off-road robots must reason about dynamics such as speed and approach angle, not only terrain geometry and semantics, to avoid getting stuck or damaged, and analytical terrain models are low fidelity. Existing off-road datasets mostly target semantic segmentation, and those with interaction data are small or have few modalities; Rellis-3D, for example, has about 20 minutes (12,000 datapoints) of trajectories.

Proposed solution

The authors teleoperated a modified Yamaha Viking ATV through tall grass, rocks, mud, puddles and steep slopes at a test site in Pittsburgh, recording throttle and steering actions, GNSS pose, stereo RGB images, stereo-derived top-down height and RGB maps, IMU, shock travel, wheel RPM and brake interventions at 10 Hz. They benchmark latent-space dynamics models with one encoder per modality fused by a product of experts, a GRU latent model and per-modality decoders, trained with state, reconstruction or contrastive losses, against a kinematic bicycle model.

Key insights

A t-SNE analysis of one-second state displacements shows that similar action sequences can lead to very different motions, so accurate prediction needs features beyond the action sequence.
The three losses (state, reconstruction, contrastive) gave similar accuracy, which the authors attribute to future observations being hard to predict in real driving, unlike simulated benchmarks with an agent on a static background.
Proprioceptive time-series data (IMU, shock travel, wheel RPM) is the most informative input, as wheel RPM is highly correlated with velocity, but camera and map inputs still add accuracy.
Exteroceptive inputs help more as terrain gets steeper and less even.

Main results

Dataset size: the abstract and contributions give roughly 200,000 interactions, while the dataset comparison table (Table I) lists 184,000 samples; it was collected as 630 trajectories, about five hours of driving.
RMSE of mean state prediction after 20 steps with the state loss: kinematic bicycle model 1.1638, image only 0.5263, image + maps 0.3521, time series 0.2176, all modalities 0.1896; the lowest overall was all modalities with the reconstruction loss at 0.1674.
Adding top-down maps to the image-only model cut prediction error by 33%; adding time-series data improved accuracy by roughly 45% from image + maps to all modalities, and adding image modalities to the time-series model by roughly 15%.
Adding exteroception to proprioception lowered error from 0.2176 to 0.1896 (13%) on the original evaluation set and from 0.7313 to 0.5394 (26%) on a separately collected, harder set.
The harder set has a median trajectory difficulty (average height change per second) of 0.2253 m/s against 0.0866 m/s for the original, and 87% of its trajectories exceed the original median.

Limitations

It remains untested whether the improved dynamics prediction is sufficient for navigation over rough terrain; integrating the models into a navigation stack with suitable cost functions and planners is left for future work.
Auxiliary prediction of sensory observations gives little benefit because of the partial observability of real-world driving.
The latent models are not trained on sequences long enough to learn any form of mapping or SLAM, which limits long-term reasoning.
Under the contrastive loss, the time-series and all-modality models perform essentially the same.
Data comes from one vehicle at one test site, driven by human teleoperation rather than random controls for safety, with brakes operated by a human driver.

Field impact

The paper was a finalist for the ICRA 2022 Outstanding Learning Paper award and has 113 citations. The dataset is released at https://github.com/castacks/tartan_drive, together with an appendix giving network architectures and training hyperparameters.

Abstract

We present TartanDrive, a large scale dataset for learning dynamics models for off-road driving. We collected a dataset of roughly 200,000 off-road driving interactions on a modified Yamaha Viking ATV with seven unique sensing modalities in diverse terrains. To the authors' knowledge, this is the largest real-world multi-modal off-road driving dataset, both in terms of number of interactions and sensing modalities. We also benchmark several state-of-the-art methods for model-based reinforcement learning from high-dimensional observations on this dataset. We find that extending these models to multi-modality leads to significant performance on off-road dynamics prediction, especially in more challenging terrains. We also identify some shortcomings with current neural network architectures for the off-road driving task. Our dataset is available at https://github.com/castacks/tartan_drive.

Open Source Robotics