DexUMI: Using Human Hand as the Universal Manipulation Interface for Dexterous Manipulation

Authors: Mengda Xu, Han Zhang, Yifan Hou, Zhenjia Xu, Linxi Fan, Manuela Veloso, Shuran Song · CoRL 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

A per-hand wearable exoskeleton plus robot-hand inpainting turns human hand demonstrations into robot-hand training data, with 86% average success across four real tasks on two hands.

Why it matters

Demonstrations are recorded without a robot present, the operator feels real contact, and the exoskeleton only allows motion the target hand can reproduce. On the tea-picking task this collected successful demonstrations 3.2 times faster than teleoperation, and the same approach worked on an underactuated hand (Inspire) and a fully actuated hand (XHand).

Problem

Transferring skills from human hands to robot hands is hard because the two differ in kinematic structure, contact surface shape, available tactile information and visual appearance, and because commercial robot hands differ widely in degrees of freedom, motor ranges and actuation. Teleoperation, the usual interface, suffers from spatial observation mismatch and gives no direct haptic feedback, while fingertip retargeting struggles with morphological differences such as thumb flexibility.

Proposed solution

DexUMI adds a hardware layer and a software layer. The hardware layer is a 3D-printed exoskeleton designed for each target hand by a bi-level optimization that matches its fingertip workspace to the robot hand's while keeping it wearable; it records joint angles with resistive encoders, wrist pose with an iPhone running ARKit, images from a 150° wide-angle camera under the wrist, and touch with the same tactile sensors as the robot hand. The software layer removes the hand and exoskeleton from the video with SAM2, fills the background with ProPainter, and composites in footage of the real robot hand replaying the same joint actions, keeping occlusions intact; a diffusion policy with a DINOv2 image encoder is trained on the result.

Key insights

Matching the exoskeleton's fingertip workspace to the robot hand, rather than to the human hand, makes recorded joint actions directly executable; the optimization moves the thumb swing joint back toward the wrist so it does not collide with the human thumb.
Relative finger actions beat absolute finger actions on every task in Table 1; the authors attribute this to a simpler, unimodal action distribution and to reactive behavior where delta actions keep accumulating until contact.
Inpainting the robot hand into the video is essential: on the Inspire hand, egg-carton success was 0.90 with inpainting, 0.10 with a green mask over the hand and 0.05 with raw exoskeleton images.
Tactile input helped only where readings were clean and vision was blocked (XHand salt picking: 0.75 with tactile vs 0.15 without); with noisy or drifting sensors it lowered success on most other tasks.
Using the same tactile sensor type and the same wrist-camera pose on exoskeleton and robot hand keeps observations consistent between data collection and deployment.

Main results

Average task success rate of 86% across four real-world tasks on two robot hands.
Inspire hand, relative finger actions with tactile and inpainting (stage-wise accumulated success, 20 episodes per task): Cube 1.00, Egg Carton 0.85, Tea tool grasp 1.00 and leaf transfer 0.85.
XHand, same configuration: Tea tool grasp 1.00 and leaf transfer 0.85; Kitchen knob 0.95, pan 0.95, salt 0.75.
Absolute finger actions with tactile failed on several stages, for example Cube 0.10 on the Inspire hand and Kitchen salt 0.00 on the XHand.
Data collection in 15-minute sessions on tea picking: 3.2 times the throughput of teleoperation, though slower than the bare human hand (Fig. 7 shows 51, 36 and 11 successful demonstrations).
Training data: 310 trajectories for Cube, 175 for Egg Carton, 400 for Tea (per hand) and 370 plus 100 knob-only trajectories for Kitchen.

Limitations

The exoskeleton design still needs hardware-specific tuning, especially for wearability, and only fingertip workspaces are matched; other contact geometry such as the palm is not modeled.
Tactile sensors were unreliable: the resistive sensors added to the Inspire hand were sensitive to how they were attached, and the XHand's electromagnetic sensors drifted after high pressure, which occurred often because the human hand applies more force than the robot.
The 3D-printed PLA-CF exoskeleton can deform under hand force without the encoders registering it, and users may not be able to fully stretch some fingers.
The software layer still needs the real robot hand to record hand images, does not fully reproduce illumination, leaves some inpainted areas blurred, and requires the camera to be rigidly fixed to the hand.
Both commercial hands showed backlash and friction, so the encoder-to-motor regression is precise in only one direction (opening or closing); the XHand mapping between motor command and fingertip position also shifts over time and after reboots.

Field impact

DexUMI was a CoRL 2025 award finalist (Best Paper / Best Student Paper) and has 184 citations. The paper refers readers to its project website, dex-umi.github.io; the paper text does not itself state a code or hardware-design release.

Abstract

We present DexUMI - a data collection and policy learning framework that uses the human hand as the natural interface to transfer dexterous manipulation skills to various robot hands. DexUMI includes hardware and software adaptations to minimize the embodiment gap between the human hand and various robot hands. The hardware adaptation bridges the kinematics gap using a wearable hand exoskeleton. It allows direct haptic feedback in manipulation data collection and adapts human motion to feasible robot hand motion. The software adaptation bridges the visual gap by replacing the human hand in video data with high-fidelity robot hand inpainting. We demonstrate DexUMI’s capabilities through comprehensive real-world experiments on two different dexterous robot hand hardware platforms, achieving an average task success rate of 86%.

Open Source Robotics