Neural Radiance Field
- Also called: NeRF, NeRFs, radiance field
- Category: Sensing › 3D maps & reconstruction
- Tags: 3d-reconstruction, computer-vision, neural-rendering, perception, deep-learning
A neural scene representation that maps a 3D position and viewing direction to color and density, trained from posed images so that new views of the scene can be rendered.
Definition
Neural Radiance Field (NeRF) is a way of representing a 3D scene with a neural network, introduced by Mildenhall et al. in 2020. The network takes a 3D point and a viewing direction as input and outputs a color and a volume density at that point. To render an image from a new camera pose, a ray is cast through each pixel, the network is evaluated at many sample points along the ray, and the colors are combined using classical volume rendering, weighted by density.
A NeRF is trained from a set of images of a scene with known camera poses, typically estimated with structure-from-motion tools such as COLMAP. Training adjusts the network so that rendered images match the real photographs. Because the rendering process is differentiable, the photographs alone provide enough supervision; no 3D ground truth is needed. A positional encoding of the input coordinates lets the network capture fine detail.
The original method was slow to train and render. Later work introduced faster scene representations, such as multiresolution hash grids and voxel grids, reducing training time from hours to minutes, and extended NeRFs to large scenes, dynamic scenes and images without known poses. 3D Gaussian Splatting, a later explicit representation, is now often chosen where real-time rendering is needed.
For robotics, NeRFs offer dense geometry and appearance from ordinary cameras. Depth can be read from the density field, semantic or feature fields can be learned alongside color, and the representation can be queried at any point. Limitations include the need for many well-posed views, artifacts in regions seen from few viewpoints, and the cost of keeping the model up to date as the scene changes.
Why it matters
NeRFs give robots a way to build detailed 3D models of scenes and objects from camera images alone, without depth sensors. They have opened new approaches to mapping, object reconstruction, grasp planning and simulation, and they are a common building block in research that links perception with manipulation and navigation.
Key applications
- 3D reconstruction of scenes and objects from camera images
- Novel-view rendering for simulation and synthetic training data
- Dense mapping for navigation and inspection
- Locating and counting objects such as fruit in agricultural scenes
- Grasp planning from reconstructed geometry
In practice
A robotics team using NeRFs typically captures a set of images with a handheld camera, a drone or the robot's own camera, recovers camera poses with COLMAP or from the robot's kinematics, and trains a model with a framework such as Nerfstudio. The trained field is then used offline, for example to extract a mesh or point cloud, render depth for grasp planning, locate or count objects, or produce realistic views for simulation and synthetic data. Because training and updating are still costly compared with conventional mapping, NeRFs are mostly used for static scenes or one-off reconstructions, and teams check reconstruction quality from viewpoints that were not used in training before trusting the geometry.
Common misconceptions
- A NeRF stores a 3D mesh — it stores a continuous function in network weights; meshes or point clouds must be extracted afterward
- NeRFs need depth sensors — they are trained from ordinary RGB images with known camera poses
- Every NeRF renders in real time — the original formulation is slow; real-time use relies on accelerated variants or other representations such as Gaussian splatting
Who should know this
- Perception engineers working on 3D reconstruction
- Robot learning researchers building simulations from real scenes
- Agricultural and inspection robotics teams
- Computer vision students
Related technologies
- 3D Gaussian Splatting
- COLMAP
- Structure from Motion
- Nerfstudio
- Volume Rendering
- Signed Distance Field
Related terms
Related projects
- LingBot-Map: Streaming 3D reconstruction from one camera: pose and depth for every frame, live
- Open3D: Modern library for 3D point cloud processing, mesh operations, and 3D machine learning
- PCL: Industry-standard C++ library for point cloud processing in robotics perception
- OpenDroneMap: Open-source photogrammetry toolkit that turns drone images into maps and 3D models
- pySLAM: Hybrid Python/C++ visual SLAM pipeline with classic and deep-learning front-ends
- OpenMind OM1: AI-native operating system for robots with modular perception, cognition, and action
Videos
- NERFs (No, not that kind) - Neural Radiance Fields - Computerphile: Computerphile