OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction

Authors: Lujie Yang, Xiaoyu Huang, Zhen Wu, Angjoo Kanazawa, Pieter Abbeel, Carmelo Sferrazza, C. Karen Liu, Rocky Duan, Guanya Shi · ICRA 2026

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Interaction-mesh retargeting with hard kinematic constraints turns human MoCap into clean humanoid references; RL policies with 5 reward terms transfer zero-shot to a Unitree G1.

Why it matters

Clean references let the authors train proprioceptive RL policies with only five reward terms, four robot domain-randomization terms and BeyondMimic's hyperparameters unchanged, with no curriculum. Effort moves from tuning rewards to producing better data, and the same pipeline adapts to other humanoids (Unitree H1, Booster T1) by changing only the keypoint correspondences and the collision model.

Problem

Humanoid skills are often learned by tracking human motion retargeted to the robot, but common retargeting pipelines (PHC, GMR, VideoMimic) rely on keypoint matching with unconstrained or softly penalized optimization. The results show foot skating, penetration and lost hand-object or foot-terrain contact, which forces heavy reward engineering in downstream RL. Collecting a separate demonstration for every object pose, object shape or terrain height is also costly.

Proposed solution

OmniRetarget builds an interaction mesh by Delaunay tetrahedralization over body keypoints plus points sampled on objects and terrain, and for each frame minimizes the Laplacian deformation between the human mesh and the robot mesh. Hard constraints enforce collision avoidance, joint and velocity limits and no sliding of stance feet, solved with a sequential SOCP warm-started from the previous frame. Re-solving with a changed object pose, object shape or terrain height turns one demonstration into many kinematically valid trajectories.

Key insights

Preserving the spatial relationships between body, object and terrain, rather than matching keypoints, removes most penetration and foot-skating artifacts.
Hard constraints avoid the tuning sensitivity of the soft collision and contact penalties used by VideoMimic.
Retargeting quality directly drives RL success; for terrain tasks, contact preservation tracked the success rate.
Kinematic augmentation widens coverage at little cost: training on the full augmented set gave 79.1% success vs 82.2% on nominal motions only, while randomizing object shape and pose during training alone performed poorly.
Building the interaction mesh in the object's local frame keeps Laplacian coordinates unchanged when the object is moved or rotated.

Main results

Robot-object (OMOMO): penetration duration 0.00 vs 0.60–0.83 for PHC, GMR and VideoMimic; maximum penetration depth 1.34 cm vs 5.11–8.50 cm; zero foot skating.
RL success on robot-object tasks: 82.20% vs PHC 71.28%, GMR 50.83% and VideoMimic 3.85%.
Robot-terrain (in-house MoCap): 94.73% RL success vs GMR 78.94%, PHC 52.63% and VideoMimic 51.75%; contact preservation 0.72 vs 0.45–0.67.
Robot-only (LAFAN1): maximum penetration 1.07 cm vs 3.22 cm for the Unitree LAFAN1 retargeting dataset, with no foot skating; both reach 100% RL success.
On hardware, a Unitree G1 runs a 30-second parkour sequence (carrying a 4.6 kg chair, climbing, jumping and rolling) and climbs a 0.9 m platform, 70% of its height, transferred zero-shot from simulation.
More than 8 hours of retargeted trajectories were generated; 39 challenging motions were used for the RL comparison.

Limitations

Retargeting is solved frame by frame; the authors suggest whole-trajectory optimization to handle noisier motion sources such as video.
Policies are proprioceptive and track references; learning autonomous visuomotor policies is left to future work.
Linearizing the collision constraints in the sequential SOCP solver leaves occasional minor penetration (maximum depth about 1.3 cm), which RL is expected to absorb.
Baseline performance may depend on hyperparameters; the authors started from the baselines' public defaults and adjusted them.
Box-moving motions share one multi-task policy, but platform climbing uses one policy per reference motion.

Field impact

Finalist for the ICRA 2026 Best Paper Award on Robot Manipulation and Locomotion, with 209 citations. The paper states that all code, the retargeted datasets (2.78 h of OMOMO box carrying, 1 h of in-house MoCap, 4.6 h of LAFAN1) and the trained policies will be publicly released.

Abstract

A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning (RL) policies. However, existing retargeting pipelines often struggle with the significant embodiment gap between humans and robots, producing physically implausible artifacts like foot-skating and penetration. More importantly, common retargeting methods neglect the rich human-object and human-environment interactions essential for expressive locomotion and loco-manipulation. To address this, we introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh that explicitly models and preserves the crucial spatial and contact relationships between an agent, the terrain, and manipulated objects. By minimizing the Laplacian deformation between the human and robot meshes while enforcing kinematic constraints, OmniRetarget generates kinematically feasible trajectories. Moreover, preserving task-relevant interactions enables efficient data augmentation, from a single demonstration to different robot embodiments, terrains, and object configurations. We comprehensively evaluate OmniRetarget by retargeting motions from OMOMO, LAFAN1, and our in-house MoCap datasets, generating over 8-hour trajectories that achieve better kinematic constraint satisfaction and contact preservation than widely used baselines. Such high-quality data enables proprioceptive RL policies to successfully execute long-horizon (up to 30 seconds) parkour and loco-manipulation skills on a Unitree G1 humanoid, trained with only 5 reward terms and simple domain randomization shared by all tasks, without any learning curriculum.

Open Source Robotics