DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation

Authors: Ruicheng Wang, Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, He Wang · ICRA 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A 1.32M-grasp ShadowHand dataset on 5355 objects, built with a faster force-closure optimiser and checked in Isaac Gym; grasp models trained on it beat those trained on DDGdata.

Why it matters

Learning-based dexterous grasp synthesis needs far more data than parallel-jaw grasping because of the hand's many degrees of freedom. DexGraspNet covers more objects, categories and grasp poses than earlier ShadowHand data, and training two existing grasp-synthesis methods on it raised both their success rate and pose diversity on both test sets.

Problem

Dexterous grasping with multi-fingered hands is far less explored than parallel-jaw grasping, partly because no large-scale dataset exists; the only ShadowHand dataset, from DDG, has 6.9K grasps on 565 objects. That data comes from the GraspIt! planner, whose search in a low-dimensional eigengrasp space gives a narrow spread of poses, while the earlier differentiable force-closure method has a low success rate and slow convergence.

Proposed solution

The authors improve differentiable force-closure grasp synthesis with an initialisation that places an opened hand facing the object from an inflated convex hull, a reverse penetration energy computed from object points so that thin meshes work, and energy terms for self-penetration and joint limits, optimised by plain gradient descent with Kaolin signed distances. Every grasp is then tested in Isaac Gym under six gravity directions with a 0.1 cm penetration limit, and only grasps passing both checks are kept.

Key insights

Starting optimisation from an opened hand that faces the object raises the success rate and speeds convergence of force-closure grasp synthesis.
Measuring penetration from object points to the hand mesh (reverse penetration energy) removes the need for object meshes with thickness, so low-quality scanned meshes can be used.
GraspIt! poses have one freedom per finger root joint and often collapse joint angles to their limits; optimisation-based synthesis gives more diverse poses (mean joint-angle entropy 5.962 vs 4.246).
Stability and diversity trade off: neither benchmarked method does well on both, and GraspTTA's higher entropy comes with a low success rate.

Main results

1.32 million grasps on 5355 objects in 133 categories, with more than 200 grasps per object; DDGdata has 6.9k grasps on 565 objects.
Generation cost: 74 min on A100 GPUs to optimise 10,000 grasps for 6000 steps with about 18% valid, vs 37 min on an NVIDIA 3090 for 512 grasps over 10,000 steps with about 3% valid for the original method; 950 GPU hours for the full dataset vs an estimated 50,000. The introduction states the same speed-up as 400 GPU hours down to 7 GPU hours per 10,000 valid grasps.
Grasp quality: mean Q1 of 0.1145 vs 0.0712 for DDGdata, and 0.2533 vs 0.2277 over the best 10%.
DDG trained on DexGraspNet vs on DDGdata: success 67.5% vs 57.4% and max penetration 0.173 vs 0.353 cm on the DexGraspNet test set; success 75.9% vs 56.4% on the DDGdata test set.
GraspTTA trained on DexGraspNet vs on DDGdata: success 24.5% vs 17.1% on the DexGraspNet test set and 39.3% vs 23.7% on the DDGdata test set.
Models trained on DexGraspNet always show higher mean joint-angle entropy, e.g. DDG at 5.683 vs 4.958 on the DexGraspNet test set.

Limitations

The dataset does not cover every grasp type in the reference taxonomy: optimisation pulls contact points toward the object, so grasps are contact-rich power grasps and precision grasps hardly appear.
The method has no semantic guidance, so functional grasps such as picking up a mug by its handle are not generated.
All grasps are for a simulated ShadowHand and validated only in Isaac Gym; no real-robot grasping is reported.
Benchmarks use an easier success test than dataset validation (one of six gravity directions, 5 mm penetration) because the original test was too strict for the baselines.
Existing grasp-synthesis methods cannot learn the dataset's full diversity while keeping a reasonable success rate.

Field impact

Finalist for the ICRA 2023 Outstanding Manipulation Paper award, with 329 citations. The paper states that the data and tools will be released on acceptance; the official project page links a public code repository.

Abstract

Object grasping using dexterous hands is a crucial yet challenging task for robotic dexterous manipulation. Compared with the field of object grasping with parallel grippers, dexterous grasping is very under-explored, partially owing to the lack of a large-scale dataset. In this work, we present a large-scale simulated dataset, DexGraspNet, for robotic dexterous grasping, along with a highly efficient synthesis method for diverse dexterous grasping synthesis. Leveraging a highly accelerated differentiable force closure estimator, we, for the first time, are able to synthesize stable and diverse grasps efficiently and robustly. We choose ShadowHand, a dexterous gripper commonly seen in robotics, and generated 1.32 million grasps for 5355 objects, covering more than 133 object categories and containing more than 200 diverse grasps for each object instance, with all grasps having been validated by the physics simulator. Compared to the previous dataset generated by GraspIt!, our dataset has not only more objects and grasps, but also higher diversity and quality. Via performing cross-dataset experiments, we show that training several algorithms of dexterous grasp synthesis on our datasets significantly outperforms training on the previous one, demonstrating the large scale and diversity of DexGraspNet. We will release the data and tools upon acceptance.

Open Source Robotics