Open Source Robotics

A curated collection of high-quality open source robotics projects, tools, and software to propel the robotics community forward.

Instance Segmentation

  • Also called: instance-level segmentation
  • Category: Sensing › Perception outputs
  • Tags: computer-vision, segmentation, perception, deep-learning, 3d-perception, manipulation

A perception task that finds each individual object in an image or point cloud and outputs a separate mask for it, usually with a class label, so touching objects of the same type stay apart.

Definition

Instance segmentation assigns each pixel of an image, or each point of a point cloud, to a specific object instance, producing one mask per object. It combines object detection, which finds and labels objects with bounding boxes, with segmentation, which works at the level of pixels or points. It is distinct from semantic segmentation, which labels every pixel with a class such as cup but does not separate two cups that touch, and from panoptic segmentation, which joins the two by giving instance masks for countable objects and class labels for background regions such as floor or wall.

Mask R-CNN, introduced by He and colleagues in 2017, became the standard two-stage approach: propose object boxes, then predict a class and a binary mask inside each one. Later methods predict masks directly from learned queries, and promptable models such as Segment Anything produce class-agnostic masks from a click or a box. In 3D, methods group points into instances by learning per-point embeddings or offsets toward object centers, or lift 2D masks from several views onto a point cloud. Performance is usually reported as mask average precision over a range of overlap thresholds, as in the COCO benchmark.

Robotics adds requirements that benchmark datasets do not stress. Robots in homes and warehouses meet objects that no training set contained, so many manipulation pipelines use unseen object instance segmentation, which separates objects by geometry and appearance without assigning a class. Masks must line up with depth so that the pixels of a mask can be turned into an object point cloud for grasping or pose estimation, and boundary errors or merges of touching objects propagate directly into failed grasps. Interactive segmentation, in which a human click or a robot's push resolves an ambiguity, is a common way to handle clutter and new categories.

Why it matters

Most manipulation pipelines need to know which pixels or points belong to which object before they can grasp, track or estimate the pose of anything. Instance segmentation is that step, and its failure modes, such as merged objects, split objects and missed novel items, are a frequent root cause of grasp failures in clutter.

Key applications

  • Bin picking and warehouse order fulfillment
  • Grasping in household clutter
  • Object pose estimation pipelines that start from a mask
  • Agricultural robots separating individual fruit or plants
  • Multi-object tracking for mobile robots and autonomous vehicles
  • Building object-level semantic maps

In practice

In a bin-picking cell, an RGB-D camera above the bin captures a frame of mixed packages. An instance segmentation network trained on the site's product range outputs one mask per package; each mask is applied to the aligned depth image to cut out that package's point cloud, which goes to a grasp planner that picks a suction point near the center of the top surface. The team tracks failures where two boxes of the same size are merged into one mask, since the grasp point then lands on the seam between them, and adds images of tightly packed boxes to the training set. For items the network has never seen, a class-agnostic model provides fallback masks, and masks with low confidence are skipped until the robot has removed neighboring items or nudged the pile.

Common misconceptions

  • Instance segmentation is the same as semantic segmentation — semantic segmentation labels classes but cannot tell two adjacent objects of the same class apart
  • A bounding box is enough for grasping — boxes include background and neighboring objects, so a mask is needed to isolate the object's own pixels or points
  • A model trained on a large benchmark will segment any object — closed-set models only know their training classes, and robots routinely meet objects outside them
  • High benchmark precision means reliable grasps — small boundary errors and merged objects in clutter matter far more to a grasp than to an average-precision score

Who should know this

  • Perception engineers working on manipulation
  • Warehouse and logistics automation developers
  • Computer vision researchers working on 3D scenes
  • Agricultural robotics engineers

Related technologies

  • Mask R-CNN
  • Segment Anything Model
  • Panoptic Segmentation
  • Unseen Object Instance Segmentation
  • COCO Dataset

Related terms

  • Point Cloud
  • Open-Vocabulary Detection
  • Semantic Map
  • Interactive Perception
  • Depth Camera
  • Grasp Planning

Related projects

  • LeRobot: HuggingFace ML library for real-world robotics in PyTorch
  • Open3D: Modern library for 3D point cloud processing, mesh operations, and 3D machine learning
  • PCL: Industry-standard C++ library for point cloud processing in robotics perception
  • Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
  • NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
  • OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation

Videos

  • Image Segmentation, Semantic Segmentation, Instance Segmentation, and Panoptic Segmentation: LearnOpenCV
  • Lecture 5.1: Fundamentals of Instance Segmentation | CVF20: UniHeidelberg

This website requires JavaScript to function properly. Please enable JavaScript to explore our collection of open source robotics projects.