Semantic Map

• Also called: semantic mapping
• Category: Sensing › 3D maps & reconstruction
• Tags: mapping, semantic-perception, embodied-ai, navigation, scene-understanding

A map that records what things are, such as object classes, instances or rooms, alongside where they are, so robots can act on goals stated in terms of objects and places.

Definition

A semantic map adds meaning to a geometric map. Besides recording where surfaces and free space are, it labels parts of the map with categories such as floor, door, chair or kitchen, and often tracks individual object instances and their positions. The labels typically come from semantic segmentation or object detection on camera images, projected into 3D using depth from an RGB-D camera, stereo or a depth prediction network, together with the robot's pose.

Semantic maps come in several forms. Top-down grids or 3D voxel grids store a class probability per cell and are common in embodied navigation and instruction-following agents. Object-level maps keep a list of objects with poses, shapes and attributes. 3D scene graphs organize the environment hierarchically into buildings, floors, rooms, places and objects, with edges for spatial relations. More recent maps store vision-language features, for example from CLIP, at each point or cell, so they can be queried with open-vocabulary text rather than a fixed label set.

Building a semantic map raises the same issues as geometric mapping, such as pose drift and loop closure, plus some of its own: segmentation errors, labels that change with viewpoint, and objects that move. Systems fuse labels over many views, for example by accumulating class probabilities per cell, and update or remove objects that are no longer observed. For a planner, the benefit is that a goal such as "put the mug in the sink" can be grounded in map coordinates, and that objects outside the current view can still be found from memory.

Why it matters

Robots increasingly receive tasks in terms of objects and places rather than coordinates. Semantic maps connect those words to locations, giving language-driven planners, household robots and inspection systems a persistent memory of what is where.

Key applications

• Object-goal and language-guided navigation
• Household mobile manipulation and object search
• Grounding natural-language instructions for task planning
• Facility inspection with labeled assets
• Change detection in warehouses and on construction sites

In practice

A mobile manipulator in a home runs an RGB-D camera, a segmentation network and a SLAM system. Each frame's per-pixel labels are projected into a voxel map using the depth image and the current pose, and the class probabilities in each voxel are updated with every new view. Detected objects are also stored as instances with a center, an extent and a last-seen time. When a user asks the robot to bring an apple, the task planner queries the map for apple instances, picks the nearest, plans a path to free space beside it, and re-detects the apple on arrival before grasping. If no apple is in the map, the planner sends the robot to explore places where apples are likely, such as the kitchen counter.

Common misconceptions

• A semantic map is just a labeled image — it is a persistent, spatially registered map that fuses labels from many views over time
• Semantic labels make geometry unnecessary — planners still need free space and collision geometry; semantics are added to the geometric map, not used instead of it
• Once mapped, objects stay where they are — objects get moved, so semantic maps need policies for updating and forgetting entries

Who should know this

• Embodied AI and navigation researchers
• Mobile manipulation engineers
• Perception and SLAM engineers
• Developers of language-driven robot planners

Related technologies

• Semantic Segmentation
• 3D Scene Graphs
• CLIP
• Kimera
• RGB-D SLAM

Related terms

• SLAM
• Topological Map
• Embodied AI
• Task Planning
• Point Cloud
• Open-Vocabulary Detection

Related projects

• GENESIS: World's fastest physics engine: 43M FPS for embodied AI learning
• LingBot-Map: Streaming 3D reconstruction from one camera: pose and depth for every frame, live
• Cartographer: Google's real-time 2D and 3D SLAM system for multi-platform robotics
• OpenDroneMap: Open-source photogrammetry toolkit that turns drone images into maps and 3D models
• XLeRobot: Practical dual-arm mobile home robot for $660 with <4hrs assembly
• RLinf: Scalable infrastructure for post-training foundation models via RL