Structure from Motion
- Also called: SfM
- Category: State estimation & navigation › Localisation
- Tags: computer-vision, 3d-reconstruction, camera-pose, photogrammetry, mapping
Recovering the camera poses and a sparse 3D point cloud of a scene from a set of overlapping images, by matching features across views and refining everything with bundle adjustment.
Definition
Structure from motion (SfM) estimates the 3D structure of a scene and the pose of the camera for each image from a collection of overlapping photographs. Nothing beyond the images is strictly required, though known camera intrinsics help. The output is a set of camera poses, often refined intrinsics and lens distortion, and a sparse point cloud of triangulated feature points. With a single moving camera the reconstruction is defined only up to an overall scale, rotation and translation, so metric scale must come from a known distance, a calibration target, GNSS or another sensor.
A typical incremental pipeline, as implemented in COLMAP (Schönberger and Frahm, 2016), detects local features such as SIFT in every image, matches them between image pairs, and keeps only matches consistent with a two-view geometric model, an essential or fundamental matrix or a homography, estimated with RANSAC. It initializes from a well-chosen image pair, then repeatedly registers a new image by solving a perspective-n-point problem against already triangulated points, triangulates new points, and runs bundle adjustment to jointly refine poses, intrinsics and points. Global SfM methods instead estimate all camera rotations and translations at once from the pairwise relations and then refine them, which is faster on large image sets but more sensitive to wrong matches.
SfM and visual SLAM solve closely related problems with the same building blocks, but SfM is usually offline and batch: images may be unordered, taken by different cameras, and processed without a time limit, which allows exhaustive matching and repeated global optimization. SLAM must run incrementally, in real time, on a sequential stream. Bundle adjustment is the optimization step inside SfM, not the whole pipeline. SfM struggles with textureless surfaces, repetitive patterns, reflective or moving objects and motion blur, all of which corrupt feature matching. Its poses are the standard starting point for dense multi-view stereo, neural radiance fields and Gaussian splatting.
Why it matters
SfM is how robotics teams turn photos or video into calibrated camera poses and a 3D model of a real place. Those poses are the input to photogrammetry, NeRF and Gaussian-splat reconstructions, and to many sim-to-real pipelines that rebuild a real scene in simulation. Knowing its assumptions explains most reconstruction failures.
Key applications
- Camera pose recovery for neural radiance fields and Gaussian splatting
- Photogrammetry and 3D scanning of objects and sites
- Rebuilding real scenes as simulation environments
- Aerial mapping and inspection with drones
- Building visual maps that robots later localize against
- Reference trajectories for evaluating visual odometry and SLAM
In practice
To build a simulation of a real room for training a mobile robot, an engineer records a slow walk-through video, extracts a few hundred sharp frames, and runs COLMAP with a single shared camera model because one camera took every frame. They inspect the result for images that failed to register, often frames facing a blank wall, and for a doubled or bent reconstruction, a sign that repeated features such as identical cabinet doors were matched to the wrong place. To fix the scale, they place a few fiducial markers of known size in the room, or measure one distance and rescale the model. The recovered poses and intrinsics then feed a neural radiance field or Gaussian-splat reconstruction that renders camera views for the robot's training.
Common misconceptions
- SfM gives metric 3D models — monocular SfM recovers geometry only up to scale; metric scale needs an external reference
- SfM and SLAM are the same thing — they share components, but SfM is typically offline over unordered images while SLAM runs incrementally in real time
- SfM is the same as bundle adjustment — bundle adjustment is the refinement step; SfM also covers feature matching, geometric verification, initialization, registration and triangulation
- SfM produces a dense model — the standard output is a sparse point cloud of matched features; dense surfaces come afterwards from multi-view stereo or neural reconstruction
Who should know this
- Perception and mapping engineers
- Researchers building NeRF or Gaussian-splat scenes
- Drone mapping and inspection teams
- Sim-to-real engineers rebuilding real environments
Related technologies
- COLMAP
- SIFT Features
- Multi-View Stereo
- OpenMVG
- Photogrammetry
- Essential Matrix
Related terms
Related projects
- LingBot-Map: Streaming 3D reconstruction from one camera: pose and depth for every frame, live
- Cartographer: Google's real-time 2D and 3D SLAM system for multi-platform robotics
- OpenDroneMap: Open-source photogrammetry toolkit that turns drone images into maps and 3D models
- RTAB-Map: Versatile graph-based SLAM supporting RGB-D, stereo, and LiDAR with real-time operation
- pySLAM: Hybrid Python/C++ visual SLAM pipeline with classic and deep-learning front-ends
- LeGO-LOAM: Classic lightweight, ground-optimized lidar odometry and mapping baseline (IROS 2018)
Videos
- Structure from Motion Problem | Structure from Motion: First Principles of Computer Vision
- Structure from Motion Octocopter - Computerphile: Computerphile