Binary Neural Network in Robotic Manipulation: Flexible Object Manipulation for Humanoid Robot Using Partially Binarized Auto-Encoder on FPGA

Authors: Satoshi Ohara, Tetsuya Ogata, Hiromitsu Awano · IROS 2021

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Binarizing only the encoder of a convolutional auto-encoder lets an end-to-end cloth-folding model for a humanoid run on a Zynq FPGA at 41.1 FPS and 3.1 W, with little accuracy loss.

Why it matters

Learned visuomotor policies usually need a GPU on or near the robot; this work shows a camera-to-joint-angle model for cloth folding running at above camera frame rate on a 3.1 W embedded board. The partial-binarization idea applies to any auto-encoder-plus-RNN controller where the decoder is only needed in training.

Problem

End-to-end learned manipulation of flexible objects such as towels has relied on GPU platforms drawing hundreds of watts, which cannot be fitted into a space-limited robot body. FPGAs are energy efficient but have little on-chip memory, and naively binarizing the image auto-encoder to shrink it corrupts the extracted image features.

Proposed solution

The authors propose a partially binarized deep convolutional auto-encoder (PB-DCAE) in which only the encoder's weights and activations are binarized, while the full-precision decoder, used only in training and then removed, lets gradients propagate. The binarized encoder runs on the FPGA programmable logic of a Xilinx ZCU102 with XNOR and popcount arithmetic, batch normalisation folded into integer thresholds and all parameters in on-chip BRAM. An LSTM that predicts the next joint angles and gripper command from image features runs in floating point on the board's ARM processor, trained with combined multi-step, single-step and internal-state losses.

Key insights

Fully binarizing the auto-encoder blocks gradient flow and corrupts features (PSNR 7.164 dB), but keeping only the training-time decoder in full precision restores quality at no inference memory cost.
The image encoder accounts for 99.94% of the multiplications per inference step, so accelerating only it on FPGA logic speeds up the whole system even with the LSTM left on the ARM core.
The LSTM is only 1.2% of the model size, so it is left unquantized to keep its accuracy.
Binarization lets all encoder parameters and feature maps sit in on-chip BRAM, avoiding external memory traffic and reducing latency and power.

Main results

Average PSNR of reconstructed images: 21.85 dB for full-precision DCAE, 7.164 dB for fully binarized DCAE and 20.78 dB for PB-DCAE.
Multi-step joint-angle prediction error: 2.98° with full-precision features, 6.32° fully binarized and 3.43° with PB-DCAE (a 0.45° degradation).
Model size fell from 51.3 MB to 2.20 MB, a 95.7% reduction.
ZCU102: 24.3 ms per step, 41.1 FPS at 3.1 W, 13.3 FPS/W; Core i7 6700K: 73.0 FPS at 55.4 W, 1.32 FPS/W; RTX 2080 Ti: 248 FPS at 69 W, 3.59 FPS/W.
Energy efficiency improved 10× over the CPU and 3.7× over the GPU while staying above the 30 FPS frame rate of a commercial RGB camera.

Limitations

Evaluation reports open-loop multi-step prediction error and image reconstruction quality on held-out demonstrations; folding success on the robot with the FPGA in the loop is not reported.
The FPGA system is slower in absolute terms than the GPU (41.1 FPS against 248 FPS) and the CPU (73.0 FPS); the gain is in energy per frame.
Only one task (cloth folding with the right gripper) on one robot is tested, with 75 demonstration sequences.
The LSTM is re-implemented in NumPy on the ARM processor and accounts for 4.48 ms of the 24.3 ms step time on the ZCU102.

Field impact

The paper was a finalist for the IROS 2021 Best RoboCup Paper Award and has 2 citations. It does not state a code or data release.

Abstract

A neural network based flexible object manipulation system for a humanoid robot on FPGA is proposed. Although the manipulations of flexible objects using robots attract ever increasing attention since these tasks are the basic and essential activities in our daily life, it has been put into practice only recently with the help of deep neural networks. However such systems have relied on GPU accelerators, which cannot be implemented into the space limited robotic body. Although field programmable gate arrays (FPGAs) are known to be energy efficient and suitable for embedded systems, the model size should be drastically reduced since FPGAs have limited on-chip memory. To this end, we propose “partially” binarized deep convolutional auto-encoder technique, where only an encoder part is binarized to compress model size without degrading the inference accuracy. The model implemented on Xilinx ZCU102 achieves 41.1 frames per second with a power consumption of 3.1 W, which corresponds to 10× and 3.7× improvements from the systems implemented on Core i7 6700K and RTX 2080 Ti, respectively.

Open Source Robotics