T-Rex
Touch-reactive policy for two dexterous hands, with open weights and a tactile dataset
License
- code: MIT
- released checkpoints and the T-Rex Dataset: MIT
- base model Qwen3-VL-2B-Instruct: Apache 2.0
- vendored Dexmate and Sharpa robot descriptions: Apache 2.0
Code: MIT, Copyright 2026 The Regents of the University of California, per LICENSE. Released checkpoints and the T-Rex Dataset: MIT, per their Hugging Face cards. Base model Qwen3-VL-2B-Instruct: Apache-2.0. Exceptions in the repository: the vendored Dexmate and Sharpa robot descriptions are Apache-2.0, and Dexmate's dexcontrol in the hardware stack is AGPL-3.0. Not included: the Sharpa Wave and Sharpa Manus SDKs that the hardware stack needs, which come under Sharpa's own terms. Checked 2026-11-08 against LICENSE, the third-party READMEs and the Hugging Face cards.
Overview
T-Rex is a vision-language-action policy for bimanual dexterous manipulation that reacts to fingertip touch while it moves, from a team at UC Berkeley, NVIDIA, Stanford, Panasonic, Sapienza University of Rome and ItalAI. A slow action expert plans a chunk of motion from camera images and a language instruction at about 5 Hz, and a small tactile expert refines that chunk at about 20 Hz from the forces and deformation measured at ten fingertips, without rerunning the vision model. The model is built on Qwen3-VL-2B and trained in three stages: egocentric human video, a 100-hour tactile robot dataset, and about 100 demonstrations per task. The release includes the pretrained and midtrained checkpoints, the post-training and inference code, a 50-hour public part of the dataset in LeRobot format, and the teleoperation stack used to record it on a Dexmate Vega-1 with two Sharpa Wave hands. The paper is presented at CoRL 2026.
T-Rex is a vision-language-action (VLA) policy for dexterous manipulation that uses touch as a fast feedback signal rather than as one more static input. Its backbone is a Mixture-of-Transformers built on the Qwen3-VL-2B vision-language model, with three transformer experts that share attention: a latent expert that predicts future visual features, an action expert that denoises a 16-step action chunk with flow matching at about 5 Hz, and a lightweight tactile expert that runs at about 20 Hz. The two action streams are cascaded: the action expert takes the first part of the denoising and caches its vision and language context, and the tactile expert finishes the chunk from that cache with fresh touch readings four times per chunk, so the policy can respond to slip or excess pressure without rerunning the vision model. Touch comes from fingertip tactile sensors in two forms: a temporal VQ-VAE turns a short window of each fingertip's 6-axis force and torque into one discrete token per finger, and a separate network reads each fingertip's deformation map.
Training runs in three stages. Following EgoScale, the latent and action experts are pretrained on 22,889 hours of egocentric human video; they are then midtrained on the 100-hour T-Rex dataset, where the tactile expert is trained from scratch; finally the policy is fine-tuned on about 100 demonstrations per task. The dataset was recorded by teleoperation on a Dexmate Vega-1 with two 22-DoF Sharpa Wave hands, each with five fingertip sensors, using Manus gloves and VIVE trackers. It pairs 22 motor primitives such as wipe, peel, insert and twist with more than 200 household objects. About 50 hours of it, 5,464 episodes, are public on Hugging Face in LeRobot v3.0 format, with head and wrist video, joint states and targets, raw and deformation tactile images and fingertip wrenches. A quickstart streams single episodes and replays them in 3D, and a browser visualizer samples the data.
The repository holds the post-training and inference code; the pretraining and midtraining code is on its full-pipeline branch. The released midtrain checkpoint is the starting point for fine-tuning on a new task, and a pretrain checkpoint is also published. Inference runs as a server on a GPU machine that the robot queries for a full pass at the start of each chunk and for tactile-only updates in between. The hardware stack adds the complete recording setup: glove and tracker teleoperation with whole-arm inverse kinematics and collision avoidance, camera and tactile streaming, synchronized episode recording, the robot-side policy client and STEP files for the wrist mount that joins each hand to the arm.
In the paper T-Rex is compared with six methods on 12 real tasks, among them turning a book page, moving an egg, applying toothpaste, opening a lock, pulling out a card and screwing in a light bulb, with 16 trials per task. It averages 65% success, against 35% for EgoScale, the strongest baseline, and 17% for π0.5; adding touch to π0.5 naively lowered it to 6%. Without touch, T-Rex itself drops from 65% to 42% in the authors' ablation over six of the tasks. These are the authors' own results on their own robot; Panasonic's announcement confirms that the paper is presented at CoRL 2026.
Why it matters
Most open VLA policies, including π0.5 from openpi, act from cameras and joint positions alone, so they cannot feel a card slipping or an egg about to crack. T-Rex brings touch into a VLA as a closed loop, and it ships everything a lab needs to try it: weights that start from tactile midtraining, the post-training code, a large public tactile dataset and the exact teleoperation stack that recorded it. Its main lesson is useful beyond this robot: in the paper, simply adding tactile input to a pretrained VLA made it worse, while running touch as a separate fast expert on top of a slow planner made it clearly better. The dataset is also a resource on its own for imitation learning research, alongside the much smaller Robotic Origami Challenge set from the same hand maker and glove-based capture such as the OSMO tactile glove. The catch is the hardware: the released weights and code assume a Vega-1 with two Sharpa Wave hands and their fingertip sensors, so most labs can use the data and the method but cannot run the checkpoints as they are.
Problem it solves
Vision-language-action policies plan from camera images at a few hertz, too slowly and too coarsely to manage the forces that decide whether a page turns, an egg stays whole or a card slides out. T-Rex adds a fast tactile expert that corrects the planned motion from fingertip force and deformation within each action chunk, and it provides the open tactile dataset and training recipe needed to learn that behaviour.
Key takeaways
- Open tactile VLA for two dexterous hands, built on Qwen3-VL-2B with latent, action and tactile experts
- A slow action expert plans at about 5 Hz and a fast tactile expert refines each chunk at about 20 Hz
- A temporal VQ-VAE turns fingertip force and torque history into one token per finger
- Trained on egocentric human video, a 100-hour tactile robot dataset and about 100 demonstrations per task
- Pretrained and midtrained checkpoints, post-training code and 50 hours of LeRobot-format tactile data are public under MIT
- Averages 65% success on 12 contact-rich tasks against 35% for the strongest baseline, in the authors' tests
Who it is for
- Robot learning researchers adding touch to VLA policies
- Labs with dexterous hands and fingertip tactile sensors, especially Sharpa Wave users
- Teams building tactile or bimanual teleoperation datasets
- Researchers studying multi-rate control and tactile representations
Limitations
- The checkpoints and the inference client are built for one platform, a Dexmate Vega-1 with two Sharpa Wave hands and their fingertip sensors; other robots need new data and post-training
- Only about 50 of the 100 dataset hours are public, and the pretraining and midtraining corpora are not released, so the full training run cannot be reproduced
- The public dataset is about 1.4 TB, and the training scripts assume nodes of 8 GPUs; the paper lists 24 NVIDIA H100 GPUs for its training runs
- All results are the authors' own, on their robot, with about 100 demonstrations per task and no outside replication yet
- The hardware stack vendors Dexmate's dexcontrol under AGPL-3.0 and needs the Sharpa Wave and Sharpa Manus SDKs, which are not part of the repository and come under Sharpa's own terms
- The Vive tracker server runs only on Windows, and the full recording setup uses four machines