Task-Oriented Grasping

• Also called: TOG, task-aware grasping
• Category: Planning & manipulation › Manipulation
• Tags: grasping, manipulation, affordances, tool-use, semantic-grasping

Choosing a grasp by what the robot will do with the object next, such as holding a knife by its handle to cut, rather than by grasp stability alone.

Definition

Most grasp planners ask one question: will this grasp hold the object? Task-oriented grasping adds a second: will this grasp let the robot carry out the intended task? A mug held by its rim may be secure but blocks pouring; a screwdriver held by its shaft cannot be turned. Task-oriented grasp selection therefore combines grasp stability with task constraints: which part of the object may be touched, which approach directions leave the functional part free, and which forces and torques the grasp must resist while the task is carried out.

Approaches have changed over time. Early analytic work described the task by the set of wrenches it would apply to the object and preferred grasps able to resist that set, as in Li and Sastry's task-oriented grasp quality measures for multifingered hands. Later work encoded object parts and their affordances, labelling regions such as handle, blade or spout, and learned which parts suit which tasks from datasets of objects annotated with task-specific grasps, such as TaskGrasp. Recent zero-shot methods generate many stable candidate grasps with a general grasp generator and then rank them using language: LERF-TOGO, for example, uses a language-embedded radiance field to find the object part named in a query and keeps grasps on that part.

Task-oriented grasping is closely tied to, but not the same as, affordance detection. An affordance map says where on an object an action could take place; task-oriented grasping is the decision of which specific grasp to execute, given the gripper, the robot's reach and the motion that follows. The task can also involve another person: in a handover, the robot often grasps so that the handle is left free and facing the receiver.

Why it matters

Robots that use objects, rather than only move them, fail if they pick an object up the wrong way and then have to regrasp or give up. Task-aware grasp choice also affects safety, for example in handing sharp tools to people.

Key applications

• Tool use with hammers, screwdrivers and spatulas
• Household and kitchen manipulation
• Safe handover of objects to people
• Pouring and manipulating containers
• Assembly where one end of a part must stay free for insertion

In practice

A kitchen assistant is asked to pass a knife to a person. Perception segments the knife and a grasp generator proposes many parallel-jaw grasps along it. A part-level step, using either a trained part segmenter or a language query over a vision-language model, separates blade from handle. For the handover the robot keeps grasps that leave the handle free and pointing towards the person; if asked to cut, it keeps grasps on the handle instead. Remaining candidates are checked for reachability and for whether the arm can perform the following motion without hitting joint limits.

Common misconceptions

• A stable grasp is a good grasp — stability is necessary but not sufficient; the grasp must also leave the object usable for the task
• Each object has one correct grasp — the right grasp depends on the task; the same knife is held differently for cutting and for handing over
• Task-oriented grasping needs task-specific training data for every object — recent zero-shot methods rank generic grasp candidates using language and vision-language models, though dedicated datasets still help with fine part distinctions

Who should know this

• Manipulation researchers working on tool use
• Grasp planning engineers
• Human-robot interaction researchers studying handovers
• Service robot developers

Related technologies

• Grasp Quality Metrics
• Part Segmentation
• Vision-Language Models
• Language Embedded Radiance Fields (LERF)
• TaskGrasp Dataset
• Grasp Pose Detection

Related terms

• Grasp Planning
• Neural Radiance Field
• VLM
• Gripper
• Affordance
• Open-Vocabulary Detection

Related projects

• LeRobot: HuggingFace ML library for real-world robotics in PyTorch
• Isaac Lab: NVIDIA robot learning framework: one task API across Newton, PhysX and Isaac Sim
• NVIDIA Isaac GR00T: NVIDIA open foundation model for humanoid robot reasoning and skills
• OpenVLA: 7B-parameter Vision-Language-Action model for robotic manipulation
• Drake: MIT robotics toolbox for model-based dynamics, optimization, and control system design
• OpenArm: 7DOF humanoid arm for physical AI research in contact-rich environments