3D Gaussian Splatting
- Also called: 3DGS, Gaussian Splatting, Gaussian splats, Gaussian radiance field
- Category: Sensing › 3D maps & reconstruction
- Tags: scene-representation, 3d-reconstruction, rendering, slam, computer-vision, simulation
A scene representation that models a 3D environment as a large set of colored, semi-transparent Gaussian blobs, which can be optimized from photos and rendered photorealistically in real time.
Definition
3D Gaussian Splatting (3DGS) is a method for reconstructing and rendering 3D scenes, introduced by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis at SIGGRAPH 2023. A scene is represented as a collection of 3D Gaussians, each with a position, a covariance that sets its size and orientation, an opacity, and a view-dependent color stored as spherical harmonic coefficients. To render an image, the Gaussians are projected onto the image plane, sorted by depth and blended front to back, a process called splatting.
The representation is learned from photographs with known camera poses, usually obtained with structure-from-motion software such as COLMAP, which also provides a sparse point cloud to initialize the Gaussians. Because the rasterizer is differentiable, the parameters of every Gaussian can be optimized by gradient descent to reduce the difference between rendered and real images. During training, Gaussians are added in under-reconstructed regions, split where they are too large, and removed when they become nearly transparent.
Compared with neural radiance fields (NeRFs), which store a scene in a neural network queried along camera rays, Gaussian splatting is an explicit representation rendered with fast tile-based rasterization instead of ray marching. This gives real-time rendering at high resolution and faster training, at the cost of higher memory use, since a scene can contain a very large number of Gaussians. Because the Gaussians are explicit, they can be moved, edited, attached to objects or deleted, which is harder with an implicit network.
Robotics uses 3DGS in several ways. As a map representation, it supports dense SLAM systems that track the camera and build a photorealistic map at the same time. As a simulation asset, a reconstructed real scene can be paired with a physics simulator so that visual policies are trained or tested on realistic images of the actual deployment environment. Limitations include reflective or transparent surfaces, geometry that is less precise than the rendered appearance suggests, and the need for good camera coverage of the scene.
Why it matters
3DGS made photorealistic reconstruction of real places fast enough to render in real time, which lets robots build realistic maps and lets teams turn a scan of a real room into visual training data. It has become a widely used bridge between real environments and simulation for vision-based robot learning.
Key applications
- Photorealistic simulation — rendering realistic camera views for training and testing visual policies
- Dense SLAM — building photorealistic maps while tracking camera pose
- Digital twins — reconstructing real workspaces, factories or homes for planning and inspection
- Drone navigation — providing realistic visual environments for vision-based flight
- Scene editing — inserting, moving or removing objects to create varied training scenes
In practice
A team typically captures a video or a set of photos of the robot's workspace, estimates camera poses with COLMAP or a similar tool, and trains a Gaussian splatting model with an open-source implementation such as the original authors' code or Nerfstudio's splatfacto. The trained scene can be rendered from any viewpoint, so they place virtual cameras matching the robot's onboard cameras to generate training images, or align the splat with a physics simulator's collision geometry so a simulated robot sees realistic views as it moves. For mapping, teams use Gaussian splatting SLAM systems with an RGB-D or monocular camera, and they validate geometry against depth sensors, because rendered images can look correct while the underlying shape is inaccurate.
Common misconceptions
- Gaussian splatting is a neural network — it is an explicit set of optimized primitives; no neural network is needed to render it
- A photorealistic render means accurate geometry — appearance can be correct while depth and surfaces are imprecise, so geometry needs separate validation
- 3DGS replaces physics simulation — it handles appearance and rendering; contact and dynamics still require a physics engine and collision geometry
- It works on any set of photos — it needs good viewpoint coverage and accurate camera poses, and struggles with reflective, transparent or moving content
Who should know this
- Perception engineers building maps and scene representations
- Robot learning researchers training vision-based policies
- Simulation engineers creating realistic environments
- Drone navigation developers
Related technologies
- Neural Radiance Fields (NeRF)
- Structure from Motion (COLMAP)
- SLAM
- Differentiable Rendering
- Point Cloud
- Nerfstudio
Related terms
Related projects
- GENESIS: World's fastest physics engine: 43M FPS for embodied AI learning
- LeRobot: HuggingFace ML library for real-world robotics in PyTorch
- LingBot-Map: Streaming 3D reconstruction from one camera: pose and depth for every frame, live
- PyBullet: Python physics simulator for robotics and RL research with fast CPU-based simulation
- oomwoo: Open-source DIY robot vacuum: 3D-printed, ROS2 + 2D LiDAR, fully local.
- Cartographer: Google's real-time 2D and 3D SLAM system for multi-platform robotics
Videos
- 3D Gaussian Splatting - Why Graphics Will Never Be The Same: IndividualKex
- 3D Gaussian Splatting! - Computerphile: Computerphile