A Deep Reinforcement Learning Environment for Particle Robot Navigation and Object Manipulation

Authors: Jeremy Shen, Erdong Xiao, Yuchen Liu, Chen Feng · ICRA 2022

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A 2D Pymunk/Gym simulator for disk-shaped particle robots with navigation and object-pushing tasks; DQN, A2C and PPO move the swarm but fall far short of a hand-crafted wave policy.

Why it matters

Particle robots trade individual capability for robustness: the paper's unresponsive-robot task shows the system continuing to operate while some robots are faulty. A shared, modifiable environment lets researchers test learned controllers for such swarms in simulation before building hardware. The benchmarks show that standard DRL algorithms do not yet solve these tasks, which gives new methods a clear baseline.

Problem

Statistical-mechanics-inspired particle robots can only expand and contract and cannot move on their own, so every task needs coordinated control of many robots. Their control has been limited to a hand-derived expansion–contraction wave policy, and deriving a new policy for each robot design and task is tedious. Existing multi-agent reinforcement learning environments use grid worlds, discretised states or game settings and cannot simulate such robots.

Proposed solution

The authors build the Deep Particle Robot (DPR) simulator on Pymunk physics and Pygame rendering with an OpenAI Gym interface; each disk-shaped robot switches between fully contracted and fully expanded, and magnets on the paddles hold neighbours together. It offers simple navigation, navigation through a gate, navigation with unresponsive robots, and object manipulation, all rewarded by displacement projected toward the goal. A centralised 'super-agent' with full observability sends one binary expand/contract action to every robot, and the tasks are benchmarked with DQN, A2C and PPO from Stable-baselines3 against the hand-crafted wave policy.

Key insights

DRL policies make the system travel about the same total distance as the wave policy but mostly in the wrong direction, which the authors read as beginning to solve self-propulsion while failing at navigation.
The hand-crafted wave policy degrades on harder tasks (75% as effective in obstacle navigation and 56% with unresponsive robots), while the DRL policies show no such clear drop.
In object manipulation the ratio of projected to net displacement of the object is above 0.97 for all DRL models, so the object is pushed mostly in the right direction even though distance lags the wave policy.
The authors attribute weak DRL performance to the low probability of the super-agent guessing a good joint action for all robots, which makes training depend on inefficient random search of the action space.

Main results

Simple navigation net displacement, as a share of the wave policy: the results section gives 10.8% (DQN), 9.1% (A2C) and 6.6% (PPO); a later sentence in the same section gives PPO's simple-navigation figure as 9.1%.
In eight of the nine DRL navigation results (three algorithms × three navigation tasks), the robots move less than one unit in the correct direction (projected displacement).
PPO's net displacement relative to the wave policy rises to 25.4% in obstacle navigation and 26.5% with unresponsive robots.
In object manipulation, DQN and PPO are 9% as effective as the wave policy and A2C is 34%, making A2C the best DRL model on that task.
Benchmarks use 2,500 steps per episode, 10,000,000 training timesteps, a [64, 128, 128, 128, 64] network and a learning rate decayed linearly from 0.0075 to 0.

Limitations

The authors conclude that current DRL algorithms are far from matching the hand-crafted control policy on these tasks.
Only the centralised super-agent setup is benchmarked; the environment supports decentralised control, but that is left for future work.
The authors note the super-agent assumption limits the system, for example the number of robots must remain constant.
The simulator is 2D; extending it to 3D and improving robot realism are planned, and paddle, friction and magnetic forces are only bounded by an inequality because the original particle-robot paper gives no definite values.
All results are in simulation with the discrete expand/contract action space; no physical particle robots are used.

Field impact

The paper was a finalist for the ICRA 2022 Outstanding Coordination Paper award and has 8 citations. The authors release the Deep Particle Robot simulator code on GitHub, linked from the project page.

Abstract

Particle robots are novel biologically-inspired robotic systems where locomotion can be achieved collectively and robustly, but not independently. While its control is currently limited to a hand-crafted policy for basic locomotion tasks, such a multi-robot system could be potentially controlled via Deep Reinforcement Learning (DRL) for different tasks more efficiently. However, the particle robot system presents a new set of challenges for DRL differing from existing swarm robotics systems: the low degrees of freedom of each robot and the increased necessity of coordination between robots. We present a 2D particle robot simulator using the OpenAI Gym interface and Pymunk as the physics engine, and introduce new tasks and challenges to research the underexplored applications of DRL in the particle robot system. Moreover, we use Stable-baselines3 to provide a set of benchmarks for the tasks. Current baseline DRL algorithms show signs of achieving the tasks but are yet unable to reach the performance of the hand-crafted policy. Further development of DRL algorithms is necessary in order to accomplish the proposed tasks.

Open Source Robotics