Physically Consistent Humanoid Loco-Manipulation Using Latent Diffusion Models

Authors: Ilyass Taouil, Haizhou Zhao, Angela Dai, Majid Khadiv · Humanoids 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

Generated images of a person doing each subtask yield hand/foot contacts and retargeted robot poses that guide whole-body trajectory optimization through long-horizon humanoid tasks.

Why it matters

The pipeline needs no task-specific heuristics and no 3D human-object interaction datasets: a high-level text plan and images of the objects are its inputs. Without the extracted contact and configuration information, the authors report that trajectory optimization could not solve either test task, so the generated images supply the long-horizon reasoning that local optimization lacks.

Problem

Humanoids are high-dimensional systems with unstable dynamics, and useful loco-manipulation tasks need long-horizon reasoning that existing planning methods do not scale to. Trajectory optimization is local and fails on such tasks without guidance, while teleoperated humanoid demonstrations are hard to collect because the robot can easily fall.

Proposed solution

From an ordered list of short text prompts and RGB-D images of the objects, a latent diffusion model (Flux) generates images of a full-body person doing each subtask. Hand and foot contacts are transferred to the 3D scene by combining semantic correspondences with a sampling-based SVD and ICP search that checks point-cloud overlap, and the person's pose (from WHAM) is retargeted to a 35-D robot configuration by inverse kinematics. These keyframes enter a centroidal-dynamics, whole-body trajectory optimization both as cost terms and as a subtask-style warm start.

Key insights

Generated 2D images of people can supply both contact locations and whole-body configurations, and the two together steer local trajectory optimization to better local minima than contacts alone.
Semantic correspondence by itself is unreliable because generated objects differ in viewpoint, shape and texture from the real ones; scoring candidate matches by point-cloud overlap (SVD, then ICP) corrects wrong contacts.
Keyframes work best as both costs and a subtask-like warm start; settings that include the warm start tend to converge better and penetrate less.
Appending a fixed description of hair and clothing to each prompt makes the image model draw a full-body person, which contact and pose extraction require.

Main results

With a minimal set of collision penalties, the full pipeline stays collision-free over the whole trajectory in both scenarios (S1: fetch a basket from a shelf by moving a box and stepping on it; S2: move a box onto a trolley and push it), while a contacts-only baseline penetrates significantly throughout (Fig. 5).
Without the extracted contact and configuration information, trajectory optimization could not solve either task; enabling all possible collision constraints instead makes it stall in local minima.
In the S1 test with a 5.0 kg box and 0 rad initial yaw, using all keyframe components gives zero penetration; removing any component either prevents convergence or gives large penetration.
Keyframe ablations cover box masses of {2.5, 5.0} kg and initial yaws of {0, 0.6, 1.2} rad for S1, and {2.5, 5.0, 7.5} kg and {0, ±0.6, ±1.2} rad for S2.
With collision penalties disabled, geometry-aware contact transfer gives substantially less penetration than geometry-unaware transfer in both scenarios.
The rigid-transform search runs for 10 iterations, and in practice the best transform is found within 3.

Limitations

Validated only in simulation (MuJoCo with a Unitree G1 model); the authors list running on a real humanoid as future work.
Assumes the high-level plan (ordered text prompts) is given, plus the target 3D location and yaw of the object for placement tasks; the authors plan to replace this with a task planner using multimodal language models.
In some ablation cases the full keyframe setting is slightly worse than partial settings, which the authors attribute to details of the trajectory optimization solver.
Camera intrinsics for generated images are set by trial and error (image resolution as focal length), and a heuristic projects contacts onto the nearest object surface to avoid penetration.
Results are given as penetration plots for two scenarios; the paper reports no success rates or planning times.

Field impact

The paper was a finalist for the Humanoids 2025 Best Oral Paper Award and has 8 citations. The paper links no code, dataset or project page; it refers to a supplementary video.

Abstract

This paper uses the capabilities of latent diffusion models (LDMs) to generate realistic RGB human-object interaction scenes to guide humanoid loco-manipulation planning. To do so, we extract from the generated images both the contact locations and robot configurations that are then used inside a whole-body trajectory optimization (TO) formulation to generate physically consistent trajectories for humanoids. We validate our full pipeline in simulation for different long-horizon loco-manipulation scenarios and perform an extensive analysis of the proposed contact and robot configuration extraction pipeline. Our results show that using the information extracted from LDMs, we can generate physically consistent trajectories that require long-horizon reasoning.

Open Source Robotics