Bundle Adjustment

• Also called: BA
• Category: State estimation & navigation › Localisation
• Tags: slam, nonlinear-optimization, computer-vision, 3d-reconstruction, state-estimation

Joint nonlinear least-squares refinement of camera poses and 3D landmark positions, and sometimes camera parameters, that minimizes the reprojection error of all observed features.

Definition

Bundle adjustment (BA) refines a 3D reconstruction by adjusting camera poses and the positions of 3D points together so that the points, projected into each image, land as close as possible to where the features were actually observed. The quantity minimized is the sum of squared reprojection errors, the pixel distances between predicted and measured feature positions, usually weighted by measurement uncertainty and wrapped in a robust loss such as Huber or Cauchy to limit the influence of wrong matches. The name refers to the bundles of light rays that connect each 3D point to the camera centers that observe it; the optimization adjusts these bundles until they agree. The technique comes from photogrammetry, and a widely cited modern treatment is the survey by Triggs, McLauchlan, Hartley and Fitzgibbon (2000).

The problem is nonlinear because of rotations and the perspective projection, so it is solved iteratively, typically with Levenberg-Marquardt. Its structure makes it tractable at scale: each residual involves only one camera and one point, so the system is sparse, and the points can be eliminated with the Schur complement to leave a much smaller system in the camera variables alone. The solution is only defined up to a gauge freedom, a global rotation and translation and, for a single camera without other sensors, a global scale, so some variables are fixed or a prior is added. Camera intrinsics and distortion can be refined at the same time when they are not well calibrated.

In robotics, BA is the core of structure from motion and of keyframe-based visual SLAM. Global BA optimizes the whole map and is run offline or after loop closures; local or sliding-window BA optimizes only the most recent keyframes and the landmarks they see, keeping older states fixed or marginalized, so it can run in real time. Visual-inertial systems extend it with IMU preintegration factors, and other sensor factors can be added in the same factor graph. BA differs from pose graph optimization in that it keeps landmarks as variables and uses the raw image measurements; a pose graph keeps only poses and relative constraints, which is faster and smaller but discards information.

Why it matters

Bundle adjustment is what turns noisy feature tracks into accurate camera trajectories and 3D maps. Most visual odometry, visual SLAM and photogrammetry pipelines rely on it, and the choice between global, local and sliding-window BA largely sets the trade-off between accuracy and real-time performance in these systems.

Key applications

• Structure from motion and photogrammetry
• Back ends of keyframe-based visual and visual-inertial SLAM
• Sliding-window visual odometry on drones, vehicles and underwater robots
• Multi-camera and camera-lidar calibration refinement
• 3D reconstruction for inspection, mapping and digital twins

In practice

In a visual-inertial SLAM system on an underwater or aerial robot, a new keyframe is created every time the camera has moved enough. A sliding-window optimizer then refines the most recent keyframe poses, their velocities and IMU biases, and all landmarks seen in those keyframes, using reprojection factors from the tracked features plus preintegrated IMU factors between keyframes. Older keyframes are marginalized into a prior so the window stays a fixed size. A Huber loss limits the effect of mismatched features caused by moving objects or reflections. When a loop closure is detected, a pose graph optimization or a full global BA corrects drift across the whole trajectory, usually in a background thread.

Common misconceptions

• Bundle adjustment is the same as pose graph optimization — BA also optimizes landmark positions against image measurements, while a pose graph keeps only poses and relative constraints
• BA fixes a bad initialization — it is a local optimizer and needs a reasonable starting estimate of poses and points to converge to the right answer
• Monocular BA recovers true scale — without another sensor or known dimension, scale is unobservable and can drift
• BA is only for offline reconstruction — local and sliding-window variants run in real time in many SLAM systems

Who should know this

• Visual SLAM and visual odometry engineers
• Computer vision and photogrammetry engineers
• State estimation engineers fusing cameras with other sensors
• Robotics students studying mapping and localization

Related technologies

• Structure from Motion
• Levenberg-Marquardt Algorithm
• Ceres Solver
• g2o
• GTSAM
• COLMAP

Related terms

• Pose Graph Optimization
• Factor Graph
• Keyframe
• SLAM
• Visual-Inertial Odometry
• Camera Intrinsics

Related projects

• LingBot-Map: Streaming 3D reconstruction from one camera: pose and depth for every frame, live
• oomwoo: Open-source DIY robot vacuum: 3D-printed, ROS2 + 2D LiDAR, fully local.
• Cartographer: Google's real-time 2D and 3D SLAM system for multi-platform robotics
• OpenDroneMap: Open-source photogrammetry toolkit that turns drone images into maps and 3D models
• FAST-LIVO2: Fast, direct LiDAR-inertial-visual odometry from HKU-MARS, published in IEEE T-RO
• DimOS: Python robotics runtime with agents, SLAM and teleop, no ROS required

Videos

• Bundle Adjustment - 5 Minutes with Cyrill: Cyrill Stachniss
• The Basics about Bundle Adjustment (Cyrill Stachniss): Cyrill Stachniss