Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method

Authors: Alvin Shek, Rui Chen, Changliu Liu · ICRA 2023

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

OPA pre-trains a graph-network policy on synthetic reach-and-avoid paths, then adapts to one human correction by updating only small per-object preference features, in under 0.5 s.

Why it matters

Separating fixed interaction dynamics from object-specific preferences lets one correction change behaviour towards one object without retraining the policy. The compact latent space can express behaviours not present in pre-training, such as arbitrary relative orientations in SO(3), and pre-training needs no human demonstrations because the data is synthetic.

Problem

Robots need to change their behaviour when a person corrects them mid-task, for example when a factory adds a new scanning step. Existing online methods either need repeated episodes of interaction or model human preferences as weights on a fixed set of hand-picked reward features, which limits what can be learned and transfers poorly to new tasks.

Proposed solution

OPA represents the scene as a graph with the robot and objects as nodes and edges from each object to the robot. Shared relation networks for position and orientation take hand-designed relative state features plus a small learned preference feature per object (1-D for position; a 1-D latent plus a quaternion rotational offset for orientation), and an aggregator sums and normalises their outputs into an action. The policy is pre-trained on synthetic Elastic Band trajectories; after a human perturbation, only the per-object features are updated by a few gradient steps on an imagined rollout, while the shared network weights stay frozen.

Key insights

Treating per-object preference features as trainable weights confines adaptation to a compact latent space: 5 gradient updates for position and 20 for orientation were enough.
Pre-training covered only four interaction classes (move closer or farther; ignore or match an object's orientation with a fixed offset), yet adaptation produced new behaviours such as arbitrary 3D orientations relative to an object.
Model structure made learned weights interpretable: learned 2D rotational offsets converged to the ground-truth offsets during training.
Unlike potential field methods, the relation network is not forced to output forces parallel to the robot–object vector, which avoids failure when an obstacle lies in the same direction as the goal.
Initialising position features for repel, attract and goal objects as 1.0, 0.5 and 0.0 made interpolation between features give interpolated behaviour, so new objects can start as 'ignore' at the midpoint.

Main results

User study with 10 CMU students on three simulated manipulator tasks, applying perturbations with computer keys.
Task 1 (carry a fragile cup low over the table): users rated OPA as understanding their preferences better, needing less effort and being slightly more predictable than a baseline that switches to gravity compensation during perturbation.
Task 2 (present a pan at a chosen orientation): average orientation error 0.2576 rad, 16% of the maximum possible π/2; satisfaction quartiles 3.5 / 4.0 / 4.0 on a five-point scale.
Task 3 (add a barcode-scanning step before the bin): 'understood my intentions' quartiles 4.0 / 4.0 / 5.0 on a five-point scale.
Online adaptation takes under 0.5 s in total; pre-training each network takes 12 minutes for 100 epochs on a laptop GPU, after 4 minutes to generate 3000 synthetic trajectories with 9 parallel processes.

Limitations

The authors state OPA relies on object-centric actions framed as reach and avoid; grasping and applying force (for example wiping a surface) are not handled.
It assumes every object can be detected, uniquely identified and its 6D pose extracted; unstructured environments are left for future work.
Results are sensitive to the adaptation learning rate and number of gradient steps: in Task 1 two users saw the robot linger near the table or ignore the goal, and in Task 3 two said the bowl should have moved closer to the scanner.
Task-centric goals that are not relative to an object, such as drawing, are not covered.
The user study used a simulated manipulator perturbed with keyboard keys, not physical contact with a real robot.

Field impact

Winner of the ICRA 2023 Outstanding Physical Human-Robot Interaction Paper award, with 7 citations. The paper links videos, code and supplementary material; the official project page points to the code repository.

Abstract

For robots to be effectively deployed in novel environments and tasks, they must be able to understand the feedback expressed by humans during intervention. This can either correct undesirable behavior or indicate additional preferences. Existing methods either require repeated episodes of interactions or assume prior known reward features, which is data-inefficient and can hardly transfer to new tasks. We relax these assumptions by describing human tasks in terms of object-centric sub-tasks and interpreting physical interventions in relation to specific objects. Our method, Object Preference Adaptation (OPA), is composed of two key stages: 1) pre-training a base policy to produce a wide variety of behaviors, and 2) online-updating only certain weights in the model according to human feedback. The key to our fast, yet simple adaptation is that general interaction dynamics between agents and objects are fixed, and only object-specific preferences are updated. Our adaptation occurs online, requires only one human intervention (one-shot), and produces new behaviors never seen during training. Trained on cheap synthetic data instead of expensive human demonstrations, our policy demonstrates impressive adaptation to human perturbations on challenging, realistic tasks in our user study. Videos, code, and supplementary material: https://alvinosaur.github.io/AboutMe/projects/opa.

Open Source Robotics