kPAM 2.0: Feedback Control for Category-Level Robotic Manipulation

Authors: Wei Gao, Russ Tedrake · ICRA 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Oriented 3D keypoints serve as the state and action of a closed-loop controller, so one policy handles 0.2 mm peg-hole insertion, LEGO, USB and wiping across shape variation.

Why it matters

Because state and action refer only to the task-relevant part of the object, the same controller works across object instances, grasp poses and initial configurations without instance-wise tuning. The framework fixes only the interfaces, so other perception modules and model-based or learned agents can be plugged in.

Problem

Closed-loop manipulation pipelines usually build feedback on a real-time 6-DOF pose estimator, but a pose defined on a fixed geometric template does not capture large shape variation within a category. The original kPAM keypoint method handles category-level pick-and-place but assumes an arbitrary rigid transformation can be applied to the object, which does not hold for contact-rich tasks such as peg-hole insertion.

Proposed solution

Keypoints are given a local orientation frame, and the robot action is expressed as the desired linear/angular velocity or force/torque of an oriented keypoint, mapped to joint commands through the keypoint Jacobian. Force/torque readings are transformed to the keypoint as a virtual sensor, inputs and outputs are re-expressed in a nominal frame to retarget to new placements, and a kPAM pick-and-place step, simplified by the oriented keypoints, brings the object to the start configuration.

Key insights

A local, task-specific oriented keypoint ignores irrelevant geometry, such as the handle the robot holds, which is what allows category-level generalization.
If the object is rigid and held tightly it can be treated as part of the robot, so standard joint-space controllers map keypoint velocity or force commands to the joints.
Transforming wrist force/torque readings to the keypoint frame makes them independent of object geometry and grasp pose.
Visual keypoint error of about 5 mm at 80 cm far exceeds a 0.2 mm clearance, so closed-loop correction with keypoints and force/torque is needed; open-loop replay fails.
Adding orientation to kPAM keypoints removes the need to solve a cost/constraint optimization for the pick-and-place target in many cases.

Main results

Failure rates, method versus open-loop baseline: whiteboard wiping 1/25 (4%) vs 16/20 (80%); 3D-printed peg-hole with 0.2 mm clearance 12/45 (26%) vs 19/20 (95%).
LEGO blocks 2/25 (8%) vs 7/20 (35%); USB port 9/20 (45%) vs 20/20 (100%).
A 6-DOF pose baseline (keypoint initialisation plus ICP to a template) failed 10/25 (40%) on wiping, 25/45 (55%) on printed peg-hole, 9/25 (36%) on LEGO and 17/20 (85%) on USB, with some trials needing a safety stop.
Wiping regulated a nominal 10 N normal force; a trial counted as failed if the eraser's front keypoint deviated more than 2 cm from the whiteboard edge or the force fell below 5 N.
For USB, about 3 mm of error on the short side causes failure; a two-step coarse-then-close wrist-camera scheme reduced perception error to 2 mm.
An agent learned by imitating successful trials worked on all three peg-hole categories but did not outperform the hand-written controller, which it almost reproduced.

Limitations

Sparse keypoints cannot handle tasks that need dense geometry, for example recognising that a peg is larger than its hole, or collision-free trajectory planning (stated by the authors).
The object is assumed static relative to the gripper, which restricts applied forces to those balanced within the static friction cones; the wrist-mounted camera cannot detect gripper–object slip (stated by the authors).
Failures came from failed grasps and from keypoint errors too large for the feedback to correct, and the USB drive had to be pre-fixed to the gripper because of its graspability (stated by the authors).
Task-specific keypoints must be chosen by the modeler for each task.
The robot model and detailed setup are referred to an Appendix that is not part of this arXiv v1 copy.

Field impact

It was a finalist for the IEEE ICRA 2021 Best Paper Award in Robot Manipulation and has 91 citations. The paper states that its video demonstration and source code are available, and gives https://sites.google.com/view/kpam2/ as the link.

Abstract

In this paper we explore generalizable, perception-to-action robotic manipulation for precise, contact-rich tasks. In particular, we contribute a framework for closed-loop robotic manipulation that automatically handles a category of objects, despite potentially unseen object instances and significant intra-category variations in shape, size and appearance. Previous approaches typically build a feedback loop on top of a realtime 6-DOF pose estimator. However, representing an object with a parameterized transformation from a fixed geometric template does not capture large intra-category shape variation. Hence we adopt the keypoint-based object representation proposed in kPAM [13] for category-level pick-and-place, and extend it to closed-loop manipulation policies with contact-rich tasks. We first augment keypoints with local orientation information. Using the oriented keypoints, we propose a novel object-centric action representation in terms of regulating the linear/angular velocity or force/torque of these oriented keypoints. This formulation is surprisingly versatile – we demonstrate that it can accomplish contact-rich manipulation tasks that require precision and dexterity for a category of objects with different shapes, sizes and appearances, such as peg-hole insertion for pegs and holes with significant shape variation and tight clearance. With the proposed object and action representation, our framework is also agnostic to the robot grasp pose and initial object configuration, making it flexible for integration and deployment. The video demonstration and source code are available on this link.

Open Source Robotics