Open-Vocabulary Detection

• Also called: open-vocabulary object detection, OVD
• Category: Sensing › Perception outputs
• Tags: object-detection, vision-language, computer-vision, perception, zero-shot

Object detection that finds objects described by arbitrary text queries, including categories never labelled during training, by matching image regions against language embeddings.

Definition

A conventional detector trained on a fixed label set can only output those labels. Open-vocabulary detection replaces the fixed classifier with a comparison between region features and text embeddings: at query time the user supplies class names or phrases, a text encoder turns them into vectors, and each candidate region is scored by its similarity to each query. Because the label space is defined by text rather than by a fixed set of output units, new categories can be queried without retraining, and the labels used at test time can differ from those seen in training.

The ability to recognize categories that have no box annotations comes from large-scale image-text pretraining. Models such as CLIP learn aligned image and text embeddings from image–caption pairs, and open-vocabulary detectors carry that alignment down to regions: by distilling CLIP features into a detector (ViLD), by attaching detection heads to a pretrained vision-language transformer (OWL-ViT), or by training on grounding data that links phrases to boxes (GLIP, Grounding DINO). The term in this sense dates from work by Zareian and colleagues in 2021. The same idea extends to segmentation and to 3D: open-vocabulary affordance detection labels the points of a point cloud with free-form affordance names by comparing per-point features with text embeddings.

Open-vocabulary detection is distinct from two related settings. Zero-shot detection usually trains on a set of seen classes and tests on a disjoint set of unseen classes, often without large image-text pretraining. Open-set detection aims to flag objects that are not in the known classes as unknown, without naming them. In robotics, open-vocabulary detectors let a system act on instructions that name arbitrary objects; VoxPoser, for example, uses one to locate the objects referred to in language-model-generated code before building its 3D value maps.

Why it matters

Robots receive instructions in natural language that name objects no fixed label set anticipated. Open-vocabulary detection lets a robot find those objects without collecting and annotating new training data for each one, which is what makes language-driven manipulation practical outside a fixed object set.

Key applications

• Language-conditioned manipulation from natural-language instructions
• Object search for mobile robots in unstructured spaces
• Building semantic maps that can be queried with text
• Labelling affordances on 3D point clouds
• Automatic labelling of robot datasets

In practice

A mobile manipulator is told to put the blue sponge in the sink. A parser extracts the phrases "blue sponge" and "sink", and an open-vocabulary detector runs on the current RGB image with those queries. The returned boxes are passed to a segmentation model, and the masks are projected into 3D with the depth image to give point clouds for grasp planning and placement. The team sets score thresholds per query rather than one global threshold, because similarity scores vary with phrasing, and confirms detections over several frames before acting.

Common misconceptions

• Open-vocabulary means any object is detected reliably — accuracy still drops for rare, small or unusually phrased categories, and results depend on the wording of the query
• Open-vocabulary detection is the same as open-set detection — open-set methods flag unfamiliar objects as unknown, while open-vocabulary methods name them from text queries
• Scores are comparable across queries — similarity scores are not calibrated between phrases, so one threshold for every query often works poorly

Who should know this

• Perception engineers building detection pipelines for robots
• Robot learning researchers working on language-conditioned policies
• Mobile manipulation developers
• Researchers building semantic maps

Related technologies

• CLIP
• OWL-ViT
• Grounding DINO
• GLIP
• ViLD
• Segment Anything (SAM)

Related terms

• VLM
• Foundation Model
• Affordance
• Task-Oriented Grasping
• Point Cloud
• RGB Camera

Related projects

• Open3D: Modern library for 3D point cloud processing, mesh operations, and 3D machine learning
• PCL: Industry-standard C++ library for point cloud processing in robotics perception
• pySLAM: Hybrid Python/C++ visual SLAM pipeline with classic and deep-learning front-ends
• OpenMind OM1: AI-native operating system for robots with modular perception, cognition, and action
• DreamZero: NVIDIA World Action Model that jointly predicts future video and robot actions
• AprilTag: The standard visual fiducial marker system for robot localization and pose estimation