What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Authors: Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín · CoRL 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Six offline learning algorithms tested on 5 simulated and 3 real manipulation tasks: history-dependent BC-RNN learns best from human demos, while batch RL struggles on human data.

Why it matters

For anyone collecting teleoperated data, the study gives concrete settings: use a history-dependent policy, include a wrist camera and pixel-shift image randomization, keep proprioceptive inputs lean, and expect checkpoint selection by validation loss to pick a weaker policy. Hyperparameters tuned in simulation were applied to a real Franka arm without real-world tuning and still produced working Lift and Can policies.

Problem

Imitation learning and batch (offline) reinforcement learning from human demonstrations lacked open-source human datasets and reproducible implementations for robot manipulation, so it was hard to judge which methods and design choices work. Human demonstrations differ from machine-generated data: they may not be Markovian, their quality varies between operators, and the training loss is only a stand-in for the task success rate used in evaluation.

Proposed solution

The authors collected teleoperated demonstrations through RoboTurk for five simulated tasks (Lift, Can, Square, Transport, Tool Hang) as single-operator Proficient-Human datasets (200 demos) and six-operator Multi-Human datasets (300 demos), plus machine-generated datasets for Lift and Can and 200-demo real-world datasets on a Franka arm for Lift, Can and Tool Hang. They trained BC, BC-RNN, HBC, BCQ, CQL and IRIS on low-dimensional and image observations, evaluated every checkpoint online in simulation, and varied observation space, hyperparameters, dataset size and the policy selection rule.

Key insights

History-dependent models (BC-RNN, HBC, IRIS) learn much better from human demonstrations than plain BC; the BC-RNN vs BC gap grows with task horizon (about 55% on Transport (PH) vs about 5% on Square (PH)) and with multi-operator data.
BCQ and CQL do well on machine-generated mixed-quality data but perform poorly on human datasets, even on a diagnostic set of paired successful and failed demonstrations from one operator.
Observation design matters: adding end-effector velocity or joint states cut low-dim agent performance by 49%–88% (relative), while removing pixel-shift randomization or the wrist camera hurt image agents.
Selecting a checkpoint by lowest validation loss or by the final epoch gave policies 10% to 100% worse than the best checkpoint, so offline policy selection remains an open problem.
Simple tasks (Lift, Can) reached 75%–100% success with 20% of the data, while complex tasks (Square, Transport) kept improving as data grew.

Main results

BC-RNN on low-dim Proficient-Human data (3 seeds): Lift 100%, Can 100%, Square 84.0%, Transport 71.3%, Tool Hang 19.3%.
BC-RNN on image observations: Square (PH) 82.0%, Transport (PH) 72.0%, Tool Hang (PH) 67.3%, against BCQ at 41.3%, 0.7% and 3.3%.
On low-dim Multi-Human data BCQ reached 62.7% on Can and 14.0% on Square, against 100% and 78.0% for BC-RNN.
On the 200-demo Square mixtures of operator quality BC-RNN scored 55.3%, 73.3% and 74.0%, below its 84.0% on the 200-demo single-operator Square (PH) dataset.
Real Franka arm, BC-RNN, final checkpoint over 30 rollouts, no real-world tuning: Lift 96.7%, Can 73.3%, Tool Hang 3.3%.
On the real Can task, removing pixel-shift randomization dropped success to 26.7% and removing the wrist camera to 43.3%.

Limitations

Simulation results report the best checkpoint found by evaluating every checkpoint online, which the authors note is not feasible on a real robot; real-world results use the final checkpoint instead.
Real-world experiments cover three tasks with 30 rollouts each and no real-world hyperparameter tuning; the authors call these results a lower bound, and the real Tool Hang policy succeeded in 3.3% of rollouts.
Machine-generated datasets exist only for Lift and Can, because the RL agent could not solve the other tasks even with substantial tuning.
The study compares six existing algorithms rather than proposing a fix; closing the batch-RL gap on human data and better offline policy selection are left to future work.
The PMLR copy refers to appendices (task details, dataset-size Tables 27 and 28, BCQ and CQL hyperparameters) that are not included in it.

Field impact

Cited 548 times (figure as given). The authors open-sourced the datasets and all algorithm implementations at the project page, giving a shared set of human-demonstration manipulation tasks and baselines for comparing offline learning methods.

Abstract

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data at https://arise-initiative.github.io/robomimic-web/

Open Source Robotics