A System for General In-Hand Object Re-Orientation

Authors: Tao Chen, Jie Xu, Pulkit Agrawal · CoRL 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Model-free RL with teacher-student distillation, lifted-pose initialisation and a gravity curriculum reorients 2000+ objects in a simulated Shadow Hand, palm up or down, without shape input.

Why it matters

The policies reach arbitrary SO(3) goal orientations without any object-shape input, which the authors call their most surprising finding and which suggests shape perception may matter less for in-hand reorientation than expected. The downward-facing and table-assisted cases cover hand configurations that prior learned reorientation work, focused on a palm-up hand, did not address.

Problem

Reorienting an object inside a multi-finger hand means controlling a high-dimensional action space while finger-object contacts keep making and breaking. Prior methods were shown mostly in simulation on simple shapes with known object models, or needed extra apparatus such as motion capture on real robots, did not generalise across objects, and almost all assumed the hand faces upward.

Proposed solution

A teacher policy is trained with PPO on privileged simulator state (including object and fingertip velocities) over more than 2000 EGAD and YCB objects, then distilled with DAgger into student policies that see either joint positions plus object pose, or only a voxelised point cloud from two RGBD cameras processed by a sparse 3D convolutional network (Sparse3D-IMPALA-Net). For a downward-facing hand without a table, episodes start from end states of a separate object-lifting policy, and gravity is introduced step by step through a curriculum.

Key insights

Policies with no access to object shape reorient many unseen objects; with the hand facing up, the learned strategy throws the object with a spin and catches it when its orientation matches the goal.
Privileged state such as object and fingertip velocities trains better policies faster, and DAgger distillation to a 31-dimensional reduced state (joint positions, object position, orientation difference) keeps most of the success rate.
With the hand facing downward and no support, direct training gives 0% success; starting episodes from the end states of a lifting policy and adding a gravity curriculum make the task learnable.
Using Chamfer distance between current and goal object point clouds as an extra success test handles symmetric objects whose different orientations look identical to the cameras.
Policies trained on 50-150 g objects reorient YCB objects re-massed to 0.3-0.5 kg zero-shot with around 75% success.

Main results

Hand facing up, full-state RNN teacher (trained and tested without domain randomisation): 95.95 ± 0.8% success on EGAD and 80.40 ± 1.6% on YCB.
Hand facing up, reduced-state RNN student: 91.96 ± 1.5% on EGAD and 81.04 ± 0.5% on YCB.
Hand facing down over a table, full-state MLP: 95.31 ± 0.9% (EGAD) and 81.59 ± 0.7% (YCB); the object-lifting policy used for initialisation succeeds 97.80% (EGAD) and 90.11% (YCB).
Hand facing down in air, reduced-state student: 74.10 ± 2.3% (EGAD) and 45.22 ± 2.1% (YCB); with the gravity curriculum YCB rises to 67.33 ± 1.9%.
Zero-shot transfer across datasets, hand up, full state: 68.82 ± 1.7% for EGAD→YCB and 96.41 ± 1.2% for YCB→EGAD.
Per-object vision policies (θ̄ = 0.2 rad, Chamfer threshold 0.01): 89.67 ± 1.2% on a mug and 84.52 ± 1.4% on a toy airplane, 50.21-66.57% on symmetric Lego Duplo pieces.

Limitations

All experiments are in simulation (a simulated Shadow Hand in Isaac Gym); the authors limited the scope because of the pandemic and give only indirect evidence for real-world transfer (domain randomisation and an analysis of peak torques).
Vision policies were trained one per object on six objects because of compute cost; one vision policy for many objects is left to future work, with GPU memory named as the main bottleneck.
Success varies strongly with object shape: long, small or thin objects cause 60% of 100 analysed YCB failures, and a further 19% end close to the goal but outside the 0.1 rad threshold.
The authors do not fully rule out that the throw-and-catch strategy exploits the simulator, although their force analysis indicates otherwise.
Reorientation with the hand facing down and no support stays well below the upward case on YCB (45.22% for the student without the curriculum, 67.33% with it).

Field impact

Winner of the CoRL 2021 Best Paper Award, with 351 citations. The paper publishes videos of the learned policies on its project page, and the official project page links a code repository (Improbable-AI/dexenv).

Abstract

In-hand object reorientation has been a challenging problem in robotics due to high dimensional actuation space and the frequent change in contact state between the fingers and the objects. We present a simple model-free framework that can learn to reorient objects with both the hand facing upwards and downwards. We demonstrate the capability of reorienting over 2000 geometrically different objects in both cases. The learned policies show strong zero-shot transfer performance on new objects. We provide evidence that these policies are amenable to real-world operation by distilling them to use observations easily available in the real world. The videos of the learned policies are available at: https://taochenshh.github.io/projects/in-hand-reorientation.

Open Source Robotics