Learning Heterogeneous Tactile Representation for General Dexterous Manipulation

Authors: Bing Han, Yichuan Peng, Haoyu Dong, Yuanmeng Hu, Yongyi Jia, Fangjing Wang, Kecheng Zheng, Yiming Zhang, Yanhong Zeng, Xiaoqing Zhang, Chen Song, Rundong Zou, Anqi Dong, Yuan Guan, Ka Leong Cheng, Xing Zhu, Yujun Shen, Nan Xue, Wei Ning · arXiv 2026

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A JEPA-trained encoder puts piezoresistive, magnetic and visuotactile sensors in one fingertip space; frozen, it lifts four policies by 13.9–24.8 points on 26 real tasks and works on an unseen sensor.

Why it matters

The encoder stays frozen and its tokens are added to an existing policy (0.13–0.16% more parameters for the three VLAs tested), so touch can be added to Diffusion Policy or a VLA without retraining the encoder. It transferred to XHand1, a magnetic hand absent from pretraining, with no parameter updates.

Problem

Commercial tactile sensors (piezoresistive, magnetic, visuotactile) each have their own channels, layout, resolution and sampling rate, so data from one cannot train a model for another. There is no large pooled tactile corpus and no shared representation, and most robots still manipulate by vision alone.

Proposed solution

LingBot-Touch places every sensor's taxels in a canonical fingertip coordinate space with five-dimensional token addresses (time, finger, 3D position) encoded by rotary embeddings, uses small per-modality MLP tokenizers and a shared Haptic Transformer, and trains it with Tactile-JEPA, which predicts the representations of masked taxels in latent space with an 80% spatial mask, a 40%-to-80% mask warm-up and layout augmentation. It is pretrained on 4,983 hours (464,431 episodes, about 1.33 billion frames) from human tactile gloves and a teleoperated dual-arm robot with Sharpa Wave hands.

Key insights

Raw tactile input helps policies at most about half as much as the learned tokens and hurts in 22 of 104 task-policy pairs.
Latent prediction scales with data while reconstruction does not: a VAE got worse from 1K to 4K hours (contact-centre error 3.03 → 4.08 mm), while LingBot-Touch improved on all five probes.
Force and dynamics transfer across sensing principles, but spatial precision depends on a sensor's own layout: at a fixed 1,000-hour budget a single-source model wins on its own sensor in 7 of 8 pairs.
Masking must suit sparse signals: an 80% mask beat 90%, and warming the ratio up from 40% avoided gradient explosions.
The Tiny model (5.46M) saturates near 4K hours while Small (21.76M) keeps improving to 5K, so capacity has to grow with the corpus.

Main results

26 real dual-arm tasks, 20 trials each: Diffusion Policy 21.6% → 35.6%, π0.5 48.1% → 69.9%, LingBot-VLA 48.3% → 73.1%, T-Rex 45.4% → 68.6% with LingBot-Touch tokens.
On XHand1 (sensor unseen in pretraining), the tokens add 13.8–24.5 points across the four policies, while raw signals change success by −1.7 to +4.6 points.
Frozen-encoder probing on Sharpa Wave: force error 0.81 N, contact-centre error 0.12 mm, tactile-to-video top-5 recall 72.6%, versus 2.31 N, 2.19 mm and 16.5% for FTP-1.
On unseen XHand1: 1.32 N, 0.16 mm and 47.1% recall, versus 3.72 N, 5.89 mm and 6.0% for FTP-1 with a sensor-specific adapter.
PPO in simulation over 12 tasks: 69.0% success with LingBot-Touch features versus 45.6% (task-trained CNN), 38.5% (raw), 30.3% (binary contact) and 21.4% (no touch).
LingBot-VLA on five Sharpa Wave tasks: 55.9% without touch, 67.5% with the FTP-1 encoder, 82.2% with LingBot-Touch.

Limitations

The authors state that real-world and simulation experiments need a wider variety of hand embodiments.
The authors note capacitive sensors are absent from pretraining because the available capacitive recordings did not meet their quality requirements.
The authors note the canonical geometry and pretraining layouts cover fingertips only, leaving palm and finger-body contacts underrepresented.
Pooling sensors costs some spatial precision compared with a single-source model given the same hours.
The tokens fall below the raw-signal configuration in 9 of 104 task-policy pairs and below the original policy in 5.

Field impact

On Hugging Face daily papers it drew 98 upvotes, in the top 5% of robotics papers there over the last six months. The paper states that inference code, a small set of sample tactile recordings and Tiny and Small checkpoints pretrained on about 5,000 hours will be released, and lists a GitHub repository and a Hugging Face checkpoint collection.

Abstract

Tactile sensing is an indispensable modality for contact-rich dexterous manipulation, a capability that hints at the ultimate frontiers of embodied intelligence. However, commercially available tactile sensors are still at a relatively early stage of development, with each type of sensor offering irreplaceable strengths alongside its own drawbacks. A key challenge is thus to free tactile understanding from sensor-specific physics, so that embodied models can effectively leverage heterogeneous tactile data. To this end, we develop LingBot-Touch, a tactile foundation model characterized by the following properties. (i) Our model yields a unified representation across different tactile sensors (i.e., piezoresistive, magnetic, and visuotactile sensors), and is easily extensible to new sensor types. (ii) The resulting tactile tokens are plug-and-play: they can be seamlessly integrated into policy models across paradigms, from diffusion policy to vision-language-action (VLA) models such as π0.5 and LingBot-VLA, consistently enhancing manipulation dexterity. (iii) Extensive experiments, spanning simulation and real-robot deployments across multiple hardware platforms, demonstrate that our tactile representation substantially benefits complex manipulation tasks while maintaining strong computational efficiency. We believe the findings in this work demonstrate the potential of leveraging cross-sensor data to scale tactile learning, and offer encouraging insights for the design of future hardware and embodied models.

Open Source Robotics