Reactive Human-To-Robot Handovers of Arbitrary Objects

Authors: Wei Yang, Chris Paxton, Arsalan Mousavian, Yu-Wei Chao, Maya Cakmak, Dieter Fox · ICRA 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Segment the hand, grasp what is left: a vision-based system lets a Franka Panda take 26 different household objects from people and re-plan when the object moves.

Why it matters

The object does not have to belong to a pre-trained detector's categories, which the authors name as the main bottleneck of the closest prior work. The closed loop lets the robot follow an object that the person turns after the robot has started moving, the cause of most failures reported in that prior work.

Problem

Assistive robots must take many kinds of objects from people, but most handover studies used a single object and only one surveyed paper used more than six. Taking an object from a hand differs from grasping a static object: the object may move, is partly hidden by fingers, limits the approach directions, and the robot's motion must feel safe.

Proposed solution

The system crops the point cloud around the hand tracked with the Azure Kinect Body Tracking SDK, segments the hand in the RGB image with a Feature Pyramid Network on ResNet-50 trained on automatically generated masks, and labels the remaining points as the object. 6-DOF GraspNet proposes grasps that are refined frame to frame by Metropolis-Hastings sampling with its grasp evaluator, grasps colliding with the hand are removed, and a Robust Logical Dynamical Systems task model with Riemannian Motion Policies picks and executes approach poses.

Key insights

Segmenting the hand, a well-defined task, and labelling the remaining cropped points as object avoids having to segment unseen objects directly.
Training masks for the hand can be produced without human annotators by cropping the tracked bare hand from the point cloud and projecting it into the image.
Refining the previous frame's grasps with the grasp evaluator keeps grasps consistent over time without retraining GraspNet; grasps are resampled when too few survive fast hand motion.
Scoring approach poses by distance to the previous best grasp and to the home pose keeps the motion smooth while tracking the person.
Rotating each grasp 180° about its Z-axis doubles the candidate pool for a two-finger gripper.

Main results

The hand segmentation model reaches 99.7% pixel accuracy, 99.5% frequency-weighted IoU and 68.0% mean IoU on a validation set collected the same way.
Systematic test with a banana, a toy mayonnaise bottle and a toy cereal box held flat, at 45° and at 90°: overall 11.39 ± 4.29 s per handover and 90% success; banana 100%, bottle 90%, cereal box 82%.
Object rotated after the robot starts moving: 0–45° gave 100% success in 11.81 ± 3.93 s; 0–90° gave 69% in 15.41 ± 7.39 s.
User study with 6 participants handing 10 objects: average approach time 10.7 ± 3.6 s, 1.3 ± 0.3 attempts and 81.8% success rate; pen (50.0%), mug (54.5%) and spoon (66.7%) were lowest.
The text (Sections V-A, V-B and VI-B) says every object was taken in the end, within three or fewer attempts in the systematic test, while Tables I and II list success rates below 100% for several objects.
Questionnaire ratings were around 4 or above in every dimension, with small variance across participants.

Limitations

Failures came from missing depth on dark surfaces, the segmentation network labelling the object as hand, and noise in the cropped point cloud from nearby objects (stated by the authors).
The cost of inverse kinematics limits how fast the robot can update its target grasp; participants asked for faster, more fluent motion (stated by the authors).
The user study had six participants, all recruited from the lab because of pandemic safety protocols (stated by the authors).
The person must hold the object in a way that leaves it graspable by the robot.
The systematic evaluation used one user and three objects, with handovers repeated until the robot had succeeded three times.

Field impact

It won the IEEE ICRA 2021 Best Paper Award on Human-Robot Interaction (HRI) and has 99 citations. The paper links a project page with further results and videos; it does not state a code or dataset release.

Abstract

Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and rigidity. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-loop motion planning with real-time, temporally-consistent grasp generation to ensure reactivity and motion smoothness. Our system is robust to different object positions and orientations, and can grasp both rigid and non-rigid objects. We demonstrate the generalizability, usability, and robustness of our approach on a novel benchmark set of 26 diverse household objects, a user study with naive users (N=6) handing over a subset of 15 objects, and a systematic evaluation examining different ways of handing objects. More results and videos can be found at https://sites.google.com/nvidia.com/handovers-of-arbitrary-objects.

Open Source Robotics