Category-Level Manipulation

• Category: Planning & manipulation › Manipulation
• Tags: manipulation, generalization, pose-estimation, 3d-perception, imitation-learning

Manipulation that generalizes across all instances of an object category, such as any mug or any drawer, including instances never seen before and without their CAD models.

Definition

Category-level manipulation aims to give a robot skills that work on any member of an object category rather than on one specific object. A robot that has learned to hang a mug by its handle should be able to hang a mug of a different shape, size and color that it has never seen, without being given a 3D model of it. This sits between instance-level manipulation, where every object has a known CAD model, and fully general manipulation of arbitrary objects.

The central difficulty is that instances in a category differ in shape, so a fixed object pose does not mean the same thing for every instance. Category-level methods therefore define a shared, task-relevant representation across the category. Examples include category-level pose in a normalized canonical frame, as in NOCS for 6D pose and size estimation; sets of semantic keypoints, such as the rim center and handle of a mug, as in kPAM; and dense learned descriptors that map corresponding points on different instances to similar features, as in Neural Descriptor Fields. A task is then written in terms of that representation, for example as a target relation between keypoints or as a gripper pose relative to the canonical frame, and transferred to a new instance by estimating the representation on it.

One attraction is data efficiency: a single demonstration on one instance can, in principle, be transferred to all others through the shared representation. The cost is that categories must be defined in advance, training needs many instances per category, usually from synthetic shape datasets, and performance degrades on unusual members whose geometry or function departs from the training set.

Why it matters

Real homes, warehouses and labs contain many variants of the same kind of object, and providing a CAD model or a new demonstration for each is impractical. Category-level methods let a skill taught once apply across a family of objects, which is a precondition for robots that work outside fixed industrial settings.

Key applications

• Household tasks with mugs, bottles, bowls and utensils
• Opening drawers, doors and other articulated objects of varying design
• Kitting and pick-and-place of product variants in warehouses
• Transferring a single demonstration to new object instances
• Task-oriented grasping of tools by their functional parts

In practice

A team wants a robot to place mugs upright on a rack. They train a category-level pose and keypoint estimator on synthetic mug models rendered with randomized textures, and record one demonstration: the gripper pose relative to the handle keypoint at grasp and the target pose of the mug's canonical frame relative to the rack peg. In operation, a depth camera segments a new mug, the network predicts its canonical pose and keypoints, the stored grasp and placement are re-expressed for this instance, and a motion planner generates the arm trajectory. Failures are logged on mugs with unusual handles, which are then added as extra shapes to the training set.

Common misconceptions

• Category-level methods still need a CAD model of each object — the aim is to handle unseen instances using only a representation learned for the whole category
• Any broad class of objects counts as a category — methods usually rely on shared geometry and function, such as mugs or laptops, and struggle with loosely defined classes like tools
• The object's center and orientation are enough to transfer a skill — because shapes differ, task-relevant parts such as handles or rims usually need their own keypoints or correspondences

Who should know this

• Manipulation researchers working on generalization
• Perception engineers building object pose estimators
• Developers of household and service robots
• Imitation learning practitioners

Related technologies

• Category-Level 6D Pose Estimation
• NOCS
• kPAM
• Neural Descriptor Fields
• Semantic Keypoints
• ShapeNet

Related terms

• Task-Oriented Grasping
• Affordance
• Articulated object
• Imitation Learning
• Point Cloud
• Equivariance

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• OpenPI: Physical Intelligence VLA foundation models for general robot control
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
• NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
• OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation
• Mobile ALOHA: Bimanual mobile manipulation with 4 robot arms on a wheeled base for teleoperation

Videos

• Lecture 14 | MIT 6.881 (Robotic Manipulation), Fall 2020 | Category-level Manipulation: underactuated