MAC-VO: Metrics-Aware Covariance for Learning-Based Stereo Visual Odometry

Authors: Yuheng Qiu, Yutian Chen, Zihao Zhang, Wenshan Wang, Sebastian Scherer · ICRA 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

MAC-VO learns per-pixel flow and depth uncertainty, converts it into full metric 3D keypoint covariances, and uses them to select keypoints and weight two-frame pose optimisation.

Why it matters

Covariances with real metric scale tell a robot how far to trust each pose estimate and give a consistent basis for fusing VO with other sensors. MAC-VO reaches its results with two-frame optimisation only, without loop closure or bundle adjustment, and transfers from synthetic training data to real datasets without fine-tuning.

Problem

Visual odometry degrades in low light, dynamic scenes and texture-less areas. Geometric VO selects keypoints by image-gradient thresholds and uses constant, hand-tuned covariances, while learned systems such as DROID-SLAM use confidence weights that are scale-agnostic and diagonal. Such weights do not reflect the actual 3D error, are inconsistent between indoor and outdoor scales, and are hard to combine with other sensors.

Proposed solution

A shared matching network, built on FlowFormer with a GMA motion aggregator and an added uncertainty decoder, predicts stereo depth, optical flow and their uncertainties; it is trained with a negative log-likelihood loss on the synthetic TartanAir dataset only. Keypoints pass an uncertainty filter (discarding pixels above 1.5× the frame's median uncertainty), a geometric filter and non-minimum suppression. The 2D uncertainties are propagated through the pinhole model into a full 3D covariance with off-diagonal terms, which weights the residuals of a two-frame pose graph solved by Levenberg-Marquardt in PyPose.

Key insights

Learned uncertainty works as a keypoint selector in place of DPVO's random selector or ORB-SLAM's hand-crafted features: it removes points on moving vehicles, occluded objects, reflective surfaces and featureless areas.
Off-diagonal covariance terms matter: dropping them (DiagCov) raised TartanAir v2 Hard translation error from 0.0141 to 0.0461 m/frame.
Scale consistency matters: a scale-agnostic (normalised) covariance raised translation error to 0.0204 m/frame on TartanAir v2 Hard and from 0.0051 to 0.0086 m/frame on the Easy set.
The depth uncertainty of a matched point is estimated as a Gaussian-weighted variance of depth in its local patch, linking flow uncertainty to depth error.
Using sparse features in the back end keeps GPU memory at 4.20 GB, 6.7 times less than DROID-SLAM (28.34 GB).

Main results

EuRoC: average translation error comparable to DROID-SLAM and rotation error about 10% better than all baselines, including SLAM systems with loop closure and global bundle adjustment.
TartanAir v2 Hard: 61.9% lower translation error than the nearest competitor; on the lunar-surface trajectory H00, 82.4% lower translation error and the lowest rotation error.
KITTI: second only to ORB-SLAM3 on translation error and 53.3% lower translation error than other VO methods; rotation error is weaker, which the authors attribute to the lack of multi-frame optimisation.
Module ablation on TartanAir v2 Hard: translation error 0.0141 m/frame for the full system versus 0.0743 with random keypoints and identity covariance, 0.0679 without covariance optimisation and 0.0188 without uncertainty-based keypoint selection.
Runtime at 640×640 on a Ryzen 9 5950X with an RTX 3090 Ti: 2.15 fps raw, 3.96 fps with TensorRT and multiprocessing, and 10.57 fps in Fast mode (Table V; the text says 10.5 fps) at 70% of full performance; Fast mode uses half precision, given as bf16 in Table V and float16 in Appendix I.

Limitations

The authors note the method only performs two-frame pose optimisation and expect gains from bundle adjustment, multi-frame optimisation and loop closure.
Rotation error on KITTI is weaker than translation error, which the authors attribute to the lack of multi-frame optimisation.
Fusing the covariance model with other sensors such as IMUs is left to future work.
The full model runs at 2.15–3.96 fps on a desktop RTX 3090 Ti; the 10.5 fps Fast mode keeps about 70% of the performance.
The depth-from-disparity approximation worsens as disparity approaches zero; Monte Carlo checks find it acceptable for error rate γ < 0.3.

Field impact

The paper won the IEEE ICRA 2025 Best Conference Paper Award and the IEEE ICRA 2025 Best Paper Award on Robot Perception, and has 37 citations. The project page named in the paper (mac-vo.github.io) links a public code repository and documentation.

Abstract

We propose the MAC-VO, a novel learning-based stereo VO that leverages the learned metrics-aware matching uncertainty for dual purposes: selecting keypoint and weighing the residual in pose graph optimization. Compared to traditional geometric methods prioritizing texture-affluent features like edges, our keypoint selector employs the learned uncertainty to filter out the low-quality features based on global inconsistency. In contrast to the learning-based algorithms that model the scale-agnostic diagonal weight matrix for covariance, we design a metrics-aware covariance model to capture the spatial error during keypoint registration and the correlations between different axes. Integrating this covariance model into pose graph optimization enhances the robustness and reliability of pose estimation, particularly in challenging environments with varying illumination, feature density, and motion patterns. On public benchmark datasets, MAC-VO outperforms existing VO algorithms and even some SLAM algorithms in challenging environments. The covariance map also provides valuable information about the reliability of the estimated poses, which can benefit decision-making for autonomous systems.

Open Source Robotics