Visual Imitation Enables Contextual Humanoid Control

Authors: Arthur Allshire, Hongsuk Choi, Junyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, Angjoo Kanazawa · CoRL 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

VideoMimic reconstructs people and scenes from phone videos, retargets the motion to a Unitree G1 and trains one policy that climbs stairs and sits or stands depending on terrain.

Why it matters

The deployed policy needs only proprioception, a LiDAR height-map and a joystick direction, and chooses between walking, stair climbing, sitting and standing from the terrain rather than from task labels. Training references come from casually recorded smartphone videos instead of a motion-capture stage, and the authors state that, to their knowledge, this is the first real-world deployment of a context-aware humanoid policy learned from monocular human videos.

Problem

Teaching humanoids context-dependent whole-body skills, such as climbing stairs or sitting on a chair, usually needs hand-tuned rewards or motion-capture data for each behavior and environment. Motion-imitation methods typically assume flat ground or instrumented capture stages, and monocular reconstruction methods often recover the person but not the surrounding scene at metric scale.

Proposed solution

From a monocular video, VideoMimic estimates per-frame SMPL human pose and 2D keypoints and a scene point cloud (MegaSaM or MonST3R), then jointly optimizes the human trajectory and the scene scale using the SMPL human height prior as the metric reference; the scene is gravity-aligned with GeoCalib and meshed with NKSR. The motion is retargeted to the Unitree G1 with joint-limit, contact and collision costs. A policy is trained in Isaac Gym in four stages: RL pre-training on LAFAN motion capture, scene-conditioned tracking with an 11 × 11 height-map, DAgger distillation to a policy conditioned only on root direction, and PPO fine-tuning with that reduced observation set.

Key insights

Optimizing human motion and scene scale together, with the SMPL body's metric height as the reference, resolves the scale ambiguity of monocular SfM point clouds.
RL pre-training on motion-capture data before tracking noisy video references is needed; without it the policy barely learns (Fig. 6), partly because reconstructed start poses can be unstable or intersect the scene.
Distilling to a policy that sees only the root direction, then running RL again with the same reduced inputs, improves on the distilled policy and lets lower-quality reference clips be used.
Two ingredients were critical for hardware deployment: looser episode-termination tolerances against the reference motion and realistic physics perturbations during training.
During staged real-world testing, conditioning on root position worked better than conditioning on root velocity.

Main results

Reconstruction on a SLOPER4D subset (Table 2): WA-MPJPE 112.13 vs 149.48 (TRAM) and 189.29 (WHAM); W-MPJPE 696.62 vs 954.90 and 1148.49; Chamfer distance 0.75 vs 10.66 for TRAM (WHAM does not recover the scene).
Joint human–scene optimization handles a 300-frame sequence in about 20 ms on an NVIDIA A100 after compilation; retargeting takes about 10 s and gravity alignment plus meshing about 60 s per 300-frame clip.
Training set of 123 self-recorded smartphone clips: 39 stair ascents, 36 descents, 13 both, 7 standing, 12 sitting, 5 sitting and standing, 11 terrain traversal; 10 flat-walking LAFAN clips make up 10% of sampling.
One policy running onboard a 23-DoF Unitree G1 at 50 Hz climbs and descends indoor and outdoor stairs, crosses steep earthen slopes and rough vegetation, and sits on and stands up from chairs and benches.
After foot slides while descending stairs, the policy recovered by briefly hopping on one leg before returning to its normal gait.

Limitations

Monocular 4D reconstruction is brittle: MegaSaM camera drift creates duplicate 'ghost' surfaces, human points get fused into the static scene, low-texture images fail, and NKSR can oversmooth narrow stair treads, so such videos are discarded.
Retargeting assumes every reference pose can be made feasible for the robot; in cluttered scenes, conflicting contact and collision costs can trap the solver in poor local minima.
The 11 × 11 LiDAR height-map is too coarse for precise contacts, manipulation or overhanging obstacles.
Scenes are modeled as a single rigid mesh, so articulated or deformable objects are not handled.
With only 123 clips the policy sometimes relies on recovery behaviors, giving jerky motion; real-robot results are shown qualitatively, without reported success rates.

Field impact

The paper received the CoRL 2025 Best Student Paper Award and has 142 citations. The authors state they will release the reconstruction code, the policy training framework and the 123-video dataset; results are shown on videomimic.net.

Abstract

How can we teach humanoids to climb staircases and sit on chairs using the surrounding environment context? Arguably, the simplest way is to just show them—casually capture a human motion video and feed it to humanoids. We introduce VIDEOMIMIC, a real-to-sim-to-real pipeline that mines everyday videos, jointly reconstructs the humans and the environment, and produces whole-body control policies for humanoid robots that perform the corresponding skills. We demonstrate the results of our pipeline on real humanoid robots, showing robust, repeatable contextual control such as staircase ascents and descents, sitting and standing from chairs and benches, as well as other dynamic whole-body skills—all from a single policy, conditioned on the environment and global root commands. VIDEOMIMIC offers a scalable path towards teaching humanoids to operate in diverse real-world environments.

Open Source Robotics