The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio

Authors: Renhao Wang, Haoran Geng, Tingle Li, Feishi Wang, Gopala Anumanchipalli, Philipp Wu, Trevor Darrell, Boyi Li, Pieter Abbeel, Jitendra Malik, Alexei A. Efros · CoRL 2025

Licences

Open Access
Paper: CC BY 4.0

Everything below draws on the full paper under its CC BY 4.0 licence; its figure is credited to the authors.

TL;DR

MultiGen pairs a physics simulator with a fine-tuned video-to-audio model to create audio-visual pouring data, training policies that pour real liquids zero-shot with no real robot data.

Why it matters

It shows a way to train robots on senses that simulators cannot model well: learn the signal from real-world video instead of simulating its physics. For pouring, where liquid is occluded or the container is transparent, the generated audio improves real-world accuracy without collecting any robot demonstrations.

Problem

Multimodal robot policies need synchronised video, audio, touch and action data, which takes precise calibration, expensive hardware and labour to collect at scale. Simulation works well for vision, but sound depends on complex wave dynamics and material interactions, so physics-based audio simulation is too expensive for large-scale robot learning. As a result, sim-to-real transfer has succeeded mainly in vision-based tasks.

Proposed solution

MultiGen runs a generative model alongside a physics simulator: RoboVerse produces motion-planned pouring trajectories under domain randomisation, and the MMAudio video-to-audio model, fine-tuned on 1,031 real pouring clips from EPIC-Kitchens and a YouTube crawl, generates audio from a virtual camera facing the receiving container. SAMv2 segmentation masks are injected as extra conditioning to bridge the gap between simulated and natural video. A diffusion policy is then trained only on this synthetic audio-visual data and deployed zero-shot on a real arm.

Key insights

A generative model can add a sensory stream that the simulator has no physical model for, provided it is conditioned on what the simulator renders.
Off-the-shelf MMAudio was inadequate: its training data holds few pouring sounds and much low-frequency Foley audio, so fine-tuning on curated real pouring clips was needed.
Conditioning the audio model on SAMv2 segmentation masks as well as RGB frames makes it more robust to the visual gap between simulation and real video.
A dedicated virtual camera in front of the receiving container keeps the audio model's input in distribution with its real-world fine-tuning videos, even when the task cameras are occluded.
Audio helps most where vision is weakest: tasks with opaque containers gained more than those with transparent ones.

Main results

Zero-shot real-world pouring with an average NMAE of 0.46, with no real robot sensory or action data used in training.
Adding generated audio cut NMAE by 23.3% on average compared with the vision-only diffusion policy.
Opaque-container tasks saw an average 29.4% NMAE reduction, transparent-container tasks 16.2%.
The vision-plus-audio policy had lower NMAE than vision-only in all eight test conditions (four opaque, four transparent), each over 12 zero-shot trials.
Compared with MANIWAV-style noise augmentation of 10 real demonstrations (1,000 samples each), MultiGen audio was more diverse (log-spectral distance) and more faithful (signal-to-distortion ratio), and policies trained on it scaled better with dataset size.

Limitations

Needs approximate alignment between simulated and real scenes (camera placement, correct robot embodiment), which still takes some manual setup (authors).
Limited generalisation to extreme container geometries; large or irregular containers cause unmodelled dynamics such as sloshing (authors).
Policies do not handle collision avoidance in cluttered scenes; obstacle-aware planning or extra perception would be needed (authors).
Demonstrated on a single task, pouring, with audio as the only generated modality.
Simulated trajectories come from a scripted motion planner with four discrete pour amounts, not from learned or human-guided demonstrations.

Field impact

Finalist for the CoRL 2025 Best Paper / Best Student Paper awards, with 10 citations so far. The licensed arXiv version (v1, titled MultiGen) does not link code, data or a project page.

Abstract

Robots must integrate multiple sensory modalities to act effectively in the real world. Yet, learning such multimodal policies at scale remains challenging. Simulation offers a viable solution, but while vision has benefited from high-fidelity simulators, other modalities (e.g. sound) can be notoriously difficult to simulate. As a result, sim-to-real transfer has succeeded primarily in vision-based tasks, with multimodal transfer still largely unrealized. In this work, we tackle these challenges by introducing MultiGen, a framework that integrates large-scale generative models into traditional physics simulators, enabling multisensory simulation. We showcase our framework on the dynamic task of robot pouring, which inherently relies on multimodal feedback. By synthesizing realistic audio conditioned on simulation video, our method enables training on rich audiovisual trajectories – without any real robot data. We demonstrate effective zero-shot transfer to real-world pouring with novel containers and liquids, highlighting the potential of generative modeling to both simulate hard-to-model modalities and close the multimodal sim-to-real gap.

Open Source Robotics