Semantic keypoint

• Also called: category-level keypoint, object keypoint
• Category: Sensing › Perception outputs
• Tags: manipulation, perception, object-representation, category-level, keypoints

A point on an object defined consistently across a whole object category, such as a mug's handle or bottom center, that a robot detects in 3D to specify and plan manipulation.

Definition

A semantic keypoint is a point on an object that has the same meaning on every instance of a category: the center of a mug's bottom, the tip of a screwdriver, the end of a peg, the toe of a shoe. Unlike a generic image feature, which is chosen because it is easy to detect and track, a semantic keypoint is chosen for what it means for a task. A perception model, usually a neural network trained on labeled images, predicts each keypoint's image location, and depth data or multiple views turn these into 3D positions.

In manipulation, semantic keypoints are an alternative to estimating a full 6-DOF object pose. A pose is only well defined relative to a fixed template model, which breaks down when instances of a category differ in size and shape: two mugs can differ in height, width and handle placement, so no single rigid transform aligns one with the other. A small set of keypoints still makes sense on both. The kPAM approach (Manuelli, Gao, Florence and Tedrake, 2019) uses this to specify tasks as costs and constraints on keypoints, for example that a mug's bottom center rests on a shelf with its top center directly above, and solves for the rigid motion of the object that satisfies them; a grasp then carries the object along that motion. kPAM 2.0 extends the approach to closed-loop feedback control, which allows contact-rich tasks such as peg-in-hole insertion.

The representation is compact and interpretable, and keypoint costs can be written directly into an optimization problem alongside collision and kinematic constraints. Its limits come from the same source. Someone has to choose which keypoints a category needs for a given task, and a few points say little about the full geometry, so collision checking usually needs a separate point cloud or mesh. Occlusion, symmetry, where several points look equivalent, and objects unlike those in the training data all degrade detection. Related representations include dense visual descriptors, which assign a learned feature to every pixel, and normalized object coordinate maps, which predict a canonical shape frame for each pixel.

Why it matters

Robots in homes, labs and warehouses meet many object instances that share a purpose but not an exact shape. Semantic keypoints make it possible to write one task specification for a whole category and still place objects precisely, without a CAD model of each object.

Key applications

• Category-level pick-and-place for objects with shape variation
• Task specification for placing, hanging and stacking
• Contact-rich assembly such as peg-in-hole insertion
• Tool use, where task-relevant points define how a tool meets a workpiece
• Tracking articulated or deformable objects through a few meaningful points

In practice

A team wants a robot to hang any of a few dozen different mugs on a rack. They annotate three keypoints on a few hundred depth images: the bottom center, the top center and a point inside the handle. A network trained on these images predicts the keypoints on new mugs, and their 3D positions are read from the depth image. The task is written as constraints: the handle point must end up on the rack hook, and the axis from bottom to top must be close to horizontal. An optimizer finds the rigid transform of the mug that satisfies them, the robot grasps the rim, and the planned motion of the mug is turned into an end-effector trajectory. Most failures happen when the handle is hidden from the camera, so the team has the robot move the camera or rotate the mug and detect again before planning.

Common misconceptions

• Semantic keypoints are the same as image feature points such as SIFT or ORB corners — those are chosen for being distinctive and trackable, while semantic keypoints are chosen for task meaning and correspond across instances
• Keypoints replace all object geometry — they describe only the task-relevant points, so collision avoidance and grasp selection still need a point cloud or mesh
• Keypoints remove pose from the problem — they avoid a template-based 6-DOF object pose, but the robot still computes a rigid transform for the object and a full pose for its gripper

Who should know this

• Manipulation engineers working with varied everyday objects
• Perception engineers training keypoint detectors
• Robot learning researchers designing object representations
• Integrators building flexible pick-and-place and assembly cells

Related technologies

• kPAM
• Keypoint Heatmap Regression
• Dense Object Descriptors
• Normalized Object Coordinate Space (NOCS)
• Perspective-n-Point

Related terms

• Category-Level Manipulation
• Affordance
• Point Cloud
• Perspective-n-Point
• Task-Oriented Grasping
• Peg-in-hole insertion

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• Open3D: Modern library for 3D point cloud processing, mesh operations, and 3D machine learning
• PCL: Industry-standard C++ library for point cloud processing in robotics perception
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
• NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
• OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation