PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-Rich Manipulation Using Tactile-Diffusion Policies

Authors: Jialiang Zhao, Naveen Kuppuswamy, Siyuan Feng, Benjamin Burchfiel, Edward Adelson · ICRA 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

PolyTouch puts camera-based touch, a contact microphone and peripheral vision in one durable finger; tactile-diffusion policies using it beat visuo-proprioceptive policies on four bimanual tasks.

Why it matters

A tactile finger that survives long contact-rich use and can be built without moulding equipment makes it practical to collect tactile data on robots at the scale learned policies need. The four test tasks were ordinary bimanual tasks not chosen for tactile sensing, and force-related failures seen without touch, such as pressing a wrench too hard or a spatula too lightly, did not occur with tactile input.

Problem

Policies that rely only on external vision and proprioception struggle with occlusion, cluttered or hard-to-see objects, and tasks that need precise regulation of contact force. Camera-based tactile sensors give high spatial resolution but typically run below 100 Hz, wear out through abrasion, tearing and delamination, need specialised equipment to fabricate, and grow bulky as the sensing area increases.

Proposed solution

PolyTouch is a robot finger that combines a camera-based tactile surface (an IMX708 camera viewing the elastomer through a curved mirror, lit by 450 nm blue LEDs and fluorescent paint), a 48 kHz piezoelectric contact microphone, and peripheral vision through two side windows, streamed over one repurposed HDMI cable to a Raspberry Pi. The default elastomer is clear 3M VHB tape wiped with aluminium flake and covered with 3M Soft & Stretch tape, in a cartridge that slides out for replacement; a silicone option uses XP-565 rubber. The tactile-diffusion policy encodes tactile and peripheral images with a pre-trained T3 encoder, wrist and scene images with CLIP, audio with an Audio Spectrogram Transformer and proprioception with an MLP, fuses them with cross-attention, and predicts actions with a U-Net diffusion policy.

Key insights

Using adhesive VHB tape as the elastomer almost entirely removes the delamination problem of silicone gels, and one VHB elastomer takes an inexperienced person under 5 minutes to make.
Excessive or insufficient force (pressing too hard when inserting a wrench, not pressing the spatula hard enough when scooping) occurred only with the vision-and-proprioception policy, never with either tactile-inclusive policy.
Imprecise grasps were seen 11 times with the visuo-proprio policy against 3 on average for tactile-inclusive policies; the authors attribute this to the close-up peripheral view during grasping.
More modalities may need more data: the gain over the visuo-proprio policy shrank when the egg-serving policy was trained on 1/3 or 2/3 of the data, which the authors link to the larger model size.

Main results

Durability under emulated tool use (10 to 30 N grasp on a spatula handle, continuous random 6D rubbing): PolyTouch-VHB ran 35 h without failure or loss of image quality; two standard GelSight Mini gels lasted 1.0 h (gel tearing) and 3.3 h (paint loss); a custom XP-565 gel on a GelSight Mini lasted 25.0 h. The abstract and introduction describe this as an at least 20-fold lifespan increase over commercial sensors.
Estimated cost about 61 USD per finger, built mostly from off-the-shelf materials without specialised equipment.
Serve Egg with all data: average task progress 81% (visuo-proprio), 88% (multi-concate), 100% (multi-crossatn); average task success 66%, 73% and 100%.
Average task success on the other tasks (visuo-proprio / multi-concate / multi-crossatn): Insert Wrench 0% / 20% / 18%, Sort Fruit 33% / 46% / 46%, Crack Egg 50% / 53% / 54%.
Blackberry-versus-blueberry sorting succeeded 80% of the time with tactile-inclusive policies against 20% with the visuo-proprio policy.
Serve Egg progress gain of multi-crossatn over visuo-proprio: 19% with all data, 3% with 2/3 and 7% with 1/3; success gain 34% with all data and 20% with 2/3.

Limitations

The viscoelastic acrylic-foam VHB tape recovers slowly after contact, and this hysteresis can cause slower response or ambiguity in dynamic tasks; the authors suggest temporal-difference images, cross-checking the peripheral view, or a less viscous material (stated limitation).
Multi-modal policies appeared to need more data to show their full benefit, possibly because of the larger model (stated by the authors).
Training sets are small, around 70 to 200 demonstrations per task, and the paper does not state how many evaluation runs each table entry is based on.
The lifespan comparison uses one emulated tool-using setup and one commercial sensor family (GelSight Mini).
The silicone elastomer responds faster than VHB but needs the same fabrication process as other silicone sensors.

Field impact

Won the IEEE ICRA 2025 Best Paper Award in Field and Service Robotics; 53 citations. The paper gives the finger's bill of materials with an estimated 61 USD per finger and its fabrication steps, and points to a project page; it links no code or design files.

Abstract

Achieving robust dexterous manipulation in unstructured domestic environments remains a significant challenge in robotics. Even with state-of-the-art robot learning methods, haptic-oblivious control strategies (i.e. those relying only on external vision and/or proprioception) often fall short due to occlusions, visual complexities, and the need for precise contact interaction control. To address these limitations, we introduce PolyTouch, a novel robot finger that integrates camera-based tactile sensing, acoustic sensing, and peripheral visual sensing into a single design that is compact and durable. PolyTouch provides high-resolution tactile feedback across multiple temporal scales, which is essential for efficiently learning complex manipulation tasks. Experiments demonstrate an at least 20-fold increase in lifespan over commercial tactile sensors, with a design that is both easy to manufacture and scalable. We then use this multi-modal tactile feedback along with visuo-proprioceptive observations to synthesize a tactile-diffusion policy from human demonstrations; the resulting contact-aware control policy significantly outperforms haptic-oblivious policies in multiple contact-aware manipulation policies. This paper highlights how effectively integrating multi-modal contact sensing can hasten the development of effective contact-aware manipulation policies, paving the way for more reliable and versatile domestic robots. More information can be found at https://polytouch.alanz.info/.

Open Source Robotics