Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks

Authors: Soroush Nasiriany, Huihan Liu, Yuke Zhu · ICRA 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

An RL agent picks among hand-coded reach, grasp, push and release primitives or a raw motor action, and sets their parameters, which cuts the exploration burden in long-horizon manipulation.

Why it matters

Robotics already has reliable modules for grasping and reaching, and this work shows a generic RL algorithm can learn to sequence them without a hand-written task plan. Because the policy acts through primitives, the same policy transferred from simulation to a real robot once the primitives were re-implemented on hardware.

Problem

Deep reinforcement learning struggles on long-horizon manipulation because the robot must explore a very large space of low-level motor actions. Hierarchical RL methods that learn skills from scratch need many samples, while task-and-motion planning and neural programming rely on domain knowledge or human supervision to compose pre-built modules.

Proposed solution

MAPLE casts control as a parameterized-action MDP: at each step a high-level task policy picks a primitive (reach, grasp, push, gripper release or a single atomic action) and a per-primitive parameter sub-network sets its inputs, such as a target position and yaw angle. Both are trained with a modified Soft Actor-Critic, and an analytic affordance score added to the reward steers primitive parameters toward relevant objects. Keeping the atomic action lets the agent handle stages, such as contact-rich insertion, that the primitives cannot express.

Key insights

A hierarchical policy, which chooses the primitive first and then its parameters, outperformed a flat policy that samples type and parameters independently
Retaining the atomic primitive was needed for Peg Insertion: the version without it reached 0% while MAPLE reached 100%
Without affordance rewards, or without the reaching or grasping primitive, the agent failed to solve Pick and Place
A Levenshtein-distance compositionality score over learned primitive sequences is lower for contact-rich tasks (Peg Insertion 0.24, Wipe 0.67) than for prehensile ones (Lift and Door 1.0)
A learned task sketch can be reused on a similar task so that only the primitive parameters need learning

Main results

Average final success across eight simulated tasks: MAPLE 90%, Atomic SAC 19%, Flat 36%, Open Loop 41%, DAC 11%, MAPLE without atomic actions 79%
The gain over the Atomic baseline is stated as a 70% increase in the introduction and as 71% higher success with 2-3× higher rewards in the results section
The paper describes MAPLE as the only method that consistently solved all single-arm tasks in the standard robosuite benchmark
Transferring the Pick and Place soda-can task sketch to a bread variant learned the target task over 5× faster than training from scratch
Real-world success over 30 trials per task: 93% on Stack (97% in simulation) and 83% on Cleanup (93% in simulation)
HIRO failed to make progress and was left out of the results

Limitations

The authors state that atomic actions can fill small gaps but not large ones that need many low-level actions, and that more work on exploration and credit assignment is needed
Affordance scores are hand-defined from object states; the authors name learned affordances as future work
Wipe remains hard: MAPLE reaches 42% success on it
In the real-world tests the pitch, yaw and table height of objects were hard coded to work around pose-estimation errors, and occlusions needed a lift-and-re-estimate routine
Real-world failures came mostly from repeated poor grasps or the robot hitting joint limits and triggering a safety stop

Field impact

Winner of the ICRA 2022 Outstanding Learning Paper award, with 155 citations. The authors release code and videos at the project page.

Abstract

Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces MAnipulation Primitive-augmented reinforcement LEarning (MAPLE), a learning framework that augments standard reinforcement learning algorithms with a pre-defined library of behavior primitives. These behavior primitives are robust functional modules specialized in achieving manipulation goals, such as grasping and pushing. To use these heterogeneous primitives, we develop a hierarchical policy that involves the primitives and instantiates their executions with input parameters. We demonstrate that MAPLE outperforms baseline approaches by a significant margin on a suite of simulated manipulation tasks. We also quantify the compositional structure of the learned behaviors and highlight our method's ability to transfer policies to new task variants and to physical hardware. Videos and code are available at https://ut-austin-rpl.github.io/maple

Open Source Robotics