GRAIL: Generating Humanoid Loco-Manipulation from World Models

Authors: Tianyi Xie, Haotian Zhang, Jinhyung Park, Zi Wang, Bowen Wen, Jiefeng Li, Xueting Li, Qingwei Ben, Haoyang Weng, Yufei Ye, David Minor, Tingwu Wang, Chenfanfu Jiang, Sanja Fidler, Jan Kautz, Linxi Fan, Yuke Zhu, Zhengyi Luo, Umar Iqbal, Ye Yuan · CoRL 2026 2026

Why it matters

Humanoid robots need demonstrations of the whole body walking, kneeling, sitting and handling objects, and today those come from teleoperating the robot or from motion capture, both of which need a physical scene and people for every new object or staircase. GRAIL replaces that with a pipeline that stays on the computer until deployment: a 3D object and scene go in, and robot-ready trajectories come out, at about 14 minutes of generation per sequence on one GPU. The authors show that this data is enough on its own: visual policies trained only on it pick up objects and climb stairs on a real Unitree G1. The code, documentation, a dataset of more than 22,000 G1 motions and tracking checkpoints are public, which makes it a working reference for anyone building synthetic humanoid data. The code is under the NVIDIA License and the dataset under CC BY-NC 4.0, so both may be used for research or evaluation only, not in a commercial product.

Problem

Humanoid loco-manipulation policies need demonstrations that cover many objects, whole-body motions and terrain shapes, and each one must be physically plausible and executable by the robot. Teleoperation and motion capture give good demonstrations but scale badly: every new object or terrain layout needs operators, instrumented actors and a rebuilt physical scene. Reconstructing robot trajectories from ordinary videos gives breadth, but camera, scale, object shape, body shape, contacts and world-space motion all have to be guessed from a single view. Earlier pipelines that use generated videos as a source of human-object interaction leave the same quantities to be inferred after generation, so their output is often not something a robot can track.

Proposed solution

GRAIL fixes the scene before it generates anything. An object asset is placed in an Infinigen room next to a human character whose proportions are fitted to the Unitree G1, settled with physics, and rendered in Blender with known camera parameters. GPT-4o writes an interaction prompt from that frame, and the Kling image-to-video model, used as a world model, generates a static-camera video of the person handling the object. Because geometry, scale, camera, scene depth and body shape are known, the reconstruction is constrained: GENMO estimates body pose with the shape held fixed, WiLoR refines the hands, a fine-tuned FoundationPose tracks the object from its known first pose, and a combined optimisation of body and object with keypoint, projection, depth and contact losses removes floating contacts, penetration and depth drift. Sequences whose object tracking disagrees with SAM2 masks are discarded. GMR retargets the motion to the G1, and two task-general trackers built on the SONIC whole-body controller turn the references into robot actions in Isaac Lab: an object-aware adaptor that adjusts SONIC's latent tokens and opens and closes the hands, and a scene-aware tracker that fine-tunes SONIC together with a network that reads a height map of the ground around the robot, for curbs, slopes, stairs and chairs. For the real robot, these trackers are distilled into egocentric RGB policies trained with domain randomisation.

Key insights

Knowing the scene before the video is generated is what makes the reconstruction usable: object geometry, camera, metric scale, scene depth and a robot-proportioned body all become inputs instead of unknowns
Human videos are the better thing to generate: current video models have stronger priors for human manipulation than for robots, and human body and hand reconstruction tools are more mature
A character pre-fitted to the robot's proportions reduces the mismatch when the motion is retargeted to the G1, so hand-object and body-scene contacts survive retargeting
Each reconstruction loss matters for the robot, not just for the picture: removing the projection, depth or contact term cuts downstream tracking success from 81.4% to between 41.6% and 53.3%
Accurate body tracking alone is not enough for manipulation: plain SONIC tracks the body best but has the lowest success, and the object-aware latent adaptor adds the object handling
Training one tracker per task family over 2,000 to 4,000 motions brings the cost to about 0.5 to 0.9 minutes of training per motion, and new motions can be added by warm-start fine-tuning

Main results

On 20 shared objects, GRAIL's generated interactions are tracked in physics simulation with 88.9% success, against 24.0% for DAViD, 15.8% for HOIDiff and 10.5% for CHOIS, with the lowest contact distance (0.008) and penetration (0.90%)
In a user study with 30 participants, GRAIL was picked for the most appropriate object use 74.7% of the time and for the most physically plausible motion 70.9% of the time, against a ceiling of 75%
On a benchmark of 124 motions across 43 objects, the task-general tracker reaches 81.4% manipulation success, against 48.5% for HDMI and 49.2% for ResMimic; without SONIC it falls to 45.0%
The pipeline produced over 20,000 sequences from 1,000 object assets and 1,000 procedural terrains, at about 14 minutes per 5-second sequence on one NVIDIA A100
On a real Unitree G1, an egocentric RGB pick-up policy trained only on GRAIL data succeeds 84% of the time over five seen objects and 80% over five unseen ones, 10 trials each
A stair-climbing policy trained only on GRAIL terrain sequences reaches 90% real-world success

Limitations

The code is under the NVIDIA License, which allows use for research or evaluation only (NVIDIA itself excepted); the released dataset is CC BY-NC 4.0, and some bundled weights and assets keep their own third-party licences, so none of it can go into a commercial product
By default, video generation calls the paid Kling API and prompts come from the OpenAI API (GPT-4o), so generating new data needs keys for outside services; a local MiniMax-H3 option was added in August 2026
The pipeline needs a 3D asset for every object and a simulator-ready scene, and the video model has to follow the requested interaction; reconstruction degrades under heavy occlusion, fast motion or object appearance that changes between frames, and filtering discards a non-trivial share of sequences
Training is heavy: each tracking policy ran 30,000 PPO iterations on 64 NVIDIA L40 GPUs, about 30 hours, and a new motion family still needs training or fine-tuning
Real-robot results cover two tasks on one robot, pick-up from a table and stair-climbing, with 10 trials per object; whole-body manipulation and sitting are shown in simulation only, and the hands use a binary open or close grasp
The released dataset covers pick-up, sitting and terrain; the whole-body manipulation data is still listed as a to-do, and the quick-start demo script is not yet released
All results are reported by the authors; at the time of writing no independent reproduction has been published

Field impact

Video generation models have mostly been used in robotics to imagine outcomes or to provide visual pretraining. GRAIL uses them as a source of physical demonstrations for a humanoid, and shows that what makes this work is deciding the 3D scene first and reconstructing inside it, rather than reading 3D motion out of an unconstrained video. With real-robot pick-up and stair-climbing trained on nothing else, it makes the case that synthetic, world-model data can sit alongside teleoperation and motion capture as a way to scale humanoid datasets, and its released dataset gives other groups a large set of G1 references for terrain and pick-up to train trackers on.

Abstract

Scaling humanoid loco-manipulation requires robot-compatible demonstrations across diverse objects, whole-body motions, and scene geometries, but teleoperation and motion capture are difficult to scale because each collection depends on physical setups, instrumented actors, and robot operation. We present GRAIL, a digital generation pipeline that remains fully virtual until deployment: it grounds video world models in 3D assets and simulator-ready scenes to synthesize interactions without rebuilding physical environments or teleoperating the robot. Rather than reconstructing unconstrained in-the-wild videos, GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction. This privileged setup better conditions 4D recovery, allowing model-based object tracking, human motion estimation, and interaction-aware optimization to reconstruct metric 4D human-object interaction (HOI) trajectories with reduced depth ambiguity and morphology mismatch. We retarget the recovered motions to a humanoid robot and train complementary task-general trackers: an object-aware latent adaptor for manipulation and a scene-aware tracker for terrain traversal. GRAIL produces over 20,000 sequences spanning pick-up, whole-body manipulation, sitting, and terrain traversal. Using only GRAIL-generated data, we train egocentric visual policies through a sim-to-real pipeline and deploy them on a Unitree G1 humanoid, achieving 84% real-world success on diverse object pick-up and 90% success on stair-climbing.

Open Source Robotics