Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning

Authors: Kun Huang, Edward S. Hu, Dinesh Jayaraman · CoRL 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

LIRF trains a second policy that acts on an outcome (pulls a door, pokes a tower, twists a valve) to tell success from failure, and uses its verdict as the reward for the task policy.

Why it matters

Task specification still needs only examples of success, with no demonstrations, engineered rewards or extra sensors, yet it works when success is not visible. The learned checks can also be reused at deployment to catch incomplete executions and let the task policy try again.

Problem

Exemplar-reward methods specify a task with images of successful outcomes, but many outcomes cannot be judged from a passive image: a loose bottle cap looks like a tight one, and a door latch can be hidden behind the door. This state aliasing makes image-based success classifiers fail in partially observable tasks.

Proposed solution

The robot is given physical 'actionable' examples of success, and an interactive reward function (IRF) policy is trained with SAC to act on these and on failure outcomes produced by an initial VICE-trained task policy, so that a single-observation classifier can tell them apart afterwards. The IRF's verdict becomes the terminal reward for training the final task policy, and the same IRF can be run after execution in a perform-verify loop until it judges the task complete.

Key insights

Interaction can reveal hidden task success: the learned IRFs pull on the door, poke the bottom block of the tower, or apply a small counter-clockwise torque to the valve.
Classifiers on single observations suffer from state aliasing, which is why VICE and GAIfO perform poorly on these partially observable tasks.
A classifier applied to one observation after the interaction is enough to label the outcome; a classifier over the full IRF trajectory is not needed.
The IRF uses no privileged information, so after training it can serve as an in-the-loop verifier.
A learned IRF was competitive with or better than hand-coded interactive checks (Manual IRF).

Main results

Task success for LIRF / LIRF+Verify (Table 1): door locking 0.640 / 0.958, block stacking 0.846 / 0.884, real-robot screwing 0.82 / 0.99.
VICE reaches 0.032, 0.310 and 0.02 on the same three tasks, and GAIfO 0.112, 0.275 and 0.0.
Ground-truth state reward gives 0.714 (door), 0.956 (blocks) and 0.87 (screwing): LIRF+Verify beats it on door locking and screwing but not on block stacking, although the text says LIRF+Verify outperforms all baselines.
IRF success-classification accuracy: 0.992 ± 0.006 for door locking, 0.852 ± 0.023 for block stacking and 0.98 for screwing (Table 2).
Actionable positive examples needed: 1 for screwing, 110 for door locking and 7k for block stacking.
Training took around 2M simulation steps for the door-locking and stacking LIRF policies, 50k steps (about 10 hours on the real robot) for screwing, and around 20k steps for each IRF policy.

Limitations

LIRF helps most when task policies trained from single-image rewards fail most of the time; in fully observable settings IRFs give marginal or no benefit (stated by the authors).
Positive examples must be stored physically and presented to the robot; this is easy for screwing (1 example) but cumbersome for block stacking (7k) (stated).
In both simulated tasks the observations 'mimic perfect vision' (object and robot poses) rather than camera images.
The real-robot test is a single valve-turning task, with screw locking mimicked by a motor under the valve.

Field impact

Winner of the CoRL 2022 Best Paper Award, with 6 citations. The paper states that all code and environments will be released, and the official project page links a public code repository.

Abstract

Physical interactions can often help reveal information that is not readily apparent. For example, we may tug at a table leg to evaluate whether it is built well, or turn a water bottle upside down to check that it is watertight. We propose to train robots to acquire such interactive behaviors automatically, for the purpose of evaluating the result of an attempted robotic skill execution. These evaluations in turn serve as "interactive reward functions" (IRFs) for training reinforcement learning policies to perform the target skill, such as screwing the table leg tightly. In addition, even after task policies are fully trained, IRFs can serve as verification mechanisms that improve online task execution. For any given task, our IRFs can be conveniently trained using only examples of successful outcomes, and no further specification is needed to train the task policy thereafter. In our evaluations on door locking and weighted block stacking in simulation, and screw tightening on a real robot, IRFs enable large performance improvements, even outperforming baselines with access to demonstrations or carefully engineered rewards. Project website: https://sites.google.com/view/lirf-corl-2022/

Open Source Robotics