APriCoT: Action Primitives based on Contact-state Transition for In-Hand Tool Manipulation

Authors: Daichi Saito, Atsushi Kanehira, Kazuhiro Sasabuchi, Naoki Wake, Jun Takamatsu, Hideki Koike, Katsushi Ikeuchi · Humanoids 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

APriCoT splits an in-hand regrasp into four short RL primitives built from detach, crossover and attach, so a simulated Shadow Hand Lite turns an object half a turn and reaches the target grasp.

Why it matters

Most learned in-hand manipulation rewards rotation alone and does not check the final grasp, which decides whether the tool can then be used. The paper shows that decomposing by contact state makes this task learnable where two published baselines failed. Because the reward is shared across primitives, new transitions can be added without new reward design.

Problem

In-hand tool manipulation must both move a tool within the hand and finish in a grasp suited to the next task, such as holding a box so its opening is clear for pouring. Learning this with RL is hard because the target grasp is reached only after many contact-state changes, which makes the reward sparse, and because the required finger motions vary widely with the contact state, forcing broad and sample-inefficient exploration.

Proposed solution

APriCoT builds a contact-state transition graph from three action representations (detach, crossover, attach), allowing only states with a prehensile grasp, at least three fingers in contact, and fingers kept in thumb-index-middle-ring order. The target task, rotating an elongated object half a turn and returning to a precision grasp, splits into four primitives that each move one finger across the object, with one policy per primitive, one shared step-wise reward, and each policy starting from the previous one's output states. Policies are trained in Isaac Gym with PPO in a teacher-student setup, where the student infers object shape from commanded and measured joint angles.

Key insights

Describing a regrasp as contact-state transitions turns one long, sparse-reward task into short primitives in which the fingers perform similar motions.
Starting and ending every primitive in the most stable state, with all fingers on the object, makes initial states easy to sample in simulation and easy to chain between policies.
Training each primitive on the whole detach-crossover-attach sequence, rather than each step on its own, avoids wasted motion such as lifting a finger too far before crossover.
The difference between commanded and measured joint angles lets the student policy encode object shape; t-SNE of its latent variables clusters by shape.

Main results

Both baselines failed the task: a modified in-hand rotation method (Baseline A) rotated the object but did not reach the desired grasp, and a method for pinchable objects (Baseline B) rolled the object during crossover and dropped it.
APriCoT completed the half-turn rotation and reached the desired precision grasp; the introduction reports a 90% success rate for objects of various shapes.
In the robustness test (1,000 sampled initial states per shape, depth 0.02-0.035 m, ε2 from ≃0 to 1), Section IV-B reports success above 90% for almost all shapes, with similar rates on shapes not used in training.
Section IV-B also reports about 75% success for thick elliptical objects (large depth, high ε2), which the authors attribute to curvature making them slip sideways.
Training ran 32,768 parallel environments in Isaac Gym on an NVIDIA RTX 3090 GPU, with 120 Hz simulation and 20 Hz control.

Limitations

All experiments are in simulation; the authors expect domain randomization and teacher-student learning to help sim-to-real transfer but did not test on hardware.
Object pose such as roll rotation cannot be detected from joint angles alone, and missing it can destabilise and drop the object; the authors plan to add visual information.
Focusing only on contact-state transitions does not guarantee a desired object orientation after manipulation.
The contact-state transition graph is designed by hand, which the authors call labour-intensive across many operations.
Only one operation (a half-turn back to a precision grasp, four primitives) is demonstrated.

Field impact

Winner of the Humanoids 2024 Best Oral Presentation Award and cited 2 times. The paper lists its reward hyperparameters, randomization ranges and network sizes. It does not state a code or model release.

Abstract

In-hand tool manipulation is an operation that not only manipulates a tool within the hand (i.e., in-hand manipulation) but also achieves a grasp suitable for a task after the manipulation. This study aims to achieve an in-hand tool manipulation skill through deep reinforcement learning. The difficulty of learning the skill arises because this manipulation requires (A) exploring long-term contact-state changes to achieve the desired grasp and (B) highly-varied motions depending on the contact-state transition. (A) leads to a sparsity of a reward on a successful grasp, and (B) requires an RL agent to explore widely within the state-action space to learn highly-varied actions, leading to sample inefficiency. To address these issues, this study proposes Action Primitives based on Contact-state Transition (APriCoT). APriCoT decomposes the manipulation into short-term action primitives by describing the operation as a contact-state transition based on three action representations (detach, crossover, attach). In each action primitive, fingers are required to perform short-term and similar actions. By training a policy for each primitive, we can mitigate the issues from (A) and (B). This study focuses on a fundamental operation as an example of in-hand tool manipulation: rotating an elongated object grasped with a precision grasp by half a turn to achieve the initial grasp. Experimental results demonstrated that ours succeeded in both the rotation and the achievement of the desired grasp, unlike existing studies. Additionally, it was found that the policy was robust to changes in object shape.

Open Source Robotics