Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation

Authors: Jared Mejia, Victoria Dean, Tess Hellebrekers, Abhinav Gupta · ICRA 2024

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Piezo contact microphones turn touch into audio, so an encoder pretrained on AudioSet video transfers to manipulation and beats vision-only and from-scratch audio policies on three real tasks.

Why it matters

Contact microphones are inexpensive compared with common tactile sensors and record vibrations at 32–48 kHz, against 30–400 Hz for optical and magnetic tactile sensors. Because their signal is audio, a team can reuse existing internet-scale audio-visual models instead of collecting large tactile datasets, which matters when only a few dozen demonstrations are available.

Problem

Large-scale pretraining is routine for visual representations in robot learning, but tactile and other non-visual modalities are still trained from scratch because there is no internet-scale data for them. In the low-data regime common in robotics, with tens of demonstrations per task, this leaves touch sensing under-used.

Proposed solution

Four piezo contact microphones on a Franka gripper record contact vibrations as audio, both at the gripper and through held tools such as a spoon or spatula. The audio encoder is initialised from AVID, a self-supervised audio-visual model pretrained on AudioSet, and the image encoder from R3M (ResNet18 pretrained on Ego4D); both stay unfrozen and are fused by a self-attention block in a behavior-cloning policy that predicts a short sequence of 6-D delta end-effector actions.

Key insights

Contact audio from a robot is close enough to internet audio that AudioSet audio-visual pretraining transfers despite the domain gap.
The scale of pretraining data is what counts: BYOL-A pretrained only on in-domain contact audio performed comparably to a randomly initialised audio encoder (better on flipping, worse on scooping and zipping).
Pretrained audio features reduced overfitting to visual details: on flipping, vision-only success dropped about 60% from train to test settings, the audio-visual policy about 20%.
Self-attention fusion is necessary but not sufficient: replacing the transformer with an MLP cut success and reward by nearly 50% on scooping, down to the level of the from-scratch baseline.
Freezing the pretrained AVID audio encoder during policy learning only slightly reduced performance.

Main results

Flipping (40 demonstrations): 50.0% success, against 25.0% for BYOL-A, 15.4% for a from-scratch audio encoder and 0.0% for vision-only.
Scooping (60 demonstrations): 78.1% success, against 50.0% from scratch, 28.1% vision-only and 25.0% BYOL-A.
Zipping (50 demonstrations): 88.9% success, against 72.2% from scratch, 66.7% BYOL-A and 44.4% vision-only.
On average 23% higher 0-1 success and 76% higher reward than the next-best baseline; outperforms or matches all baselines in 8 of 9 task configurations.
On scooping, success rose steadily from 30 to 60 to 90 demonstrations, at roughly the same rate as the vision-only baseline.
The policy has around 20M parameters and trains in 1–1.5 hours on a single RTX 3080 Ti GPU.

Limitations

The authors note contact microphones may be less useful in less dynamic tasks such as pick and place.
They may also be less useful when the robot itself generates significant vibrations.
They may not help with deformable objects that do not emit perceptible vibrations on contact.
Evaluation covers three tasks on one robot arm with at most 60 demonstrations per task (90 in the scaling ablation).

Field impact

Finalist for the ICRA 2024 IEEE ICRA Best Paper Award in Robot Manipulation, with 31 citations. The paper points to a project page with videos of the real-robot experiments; it does not link code.

Abstract

Although pre-training on a large amount of data is beneficial for robot learning, current paradigms only perform large-scale pretraining for visual representations, whereas representations for other modalities are trained from scratch. In contrast to the abundance of visual data, it is unclear what relevant internet-scale data may be used for pretraining other modalities such as tactile sensing. Such pretraining becomes increasingly crucial in the low-data regimes common in robotics applications. In this paper, we address this gap by using contact microphones as an alternative tactile sensor. Our key insight is that contact microphones capture inherently audio-based information, allowing us to leverage large-scale audio-visual pretraining to obtain representations that boost the performance of robotic manipulation. To the best of our knowledge, our method is the first approach leveraging large-scale multisensory pre-training for robotic manipulation. For supplementary information including videos of real robot experiments, please see https://sites.google.com/view/hearing-touch.

Open Source Robotics