TL;DR
MMAudio is a novel multimodal joint training framework that synthesizes high-quality, perfectly synchronized audio from video and optional text inputs. By training a single transformer architecture from scratch on both limited audio-visual data and large-scale audio-text data, it overcomes data scarcity bottlenecks. The resulting model achieves state-of-the-art performance in video-to-audio synthesis, generating an 8-second clip in just 1.23 seconds using a compact 157M parameter network.
What Problem Does This Paper Address?
Generating realistic ambient sound and sound effects for a video—a process known as Foley synthesis—requires two main ingredients: semantic alignment (e.g., a video of rain must sound like rain, not a crackling fire) and temporal alignment (e.g., the sound of a drumbeat must precisely match the visual impact within milliseconds).
Current state-of-the-art methods struggle with a massive data bottleneck. High-quality, clean audio-visual datasets are incredibly difficult to curate because most internet videos contain background music or human speech, which are unhelpful for training a general Foley model. Consequently, standard datasets like VGGSound offer only about 550 hours of video. To bypass this, previous works tried grafting complex "control modules" onto frozen, pretrained text-to-audio models. However, this complicates network architectures and restricts the model's flexibility.
MMAudio tackles this by proposing a unified framework that can be trained from scratch simultaneously on both scarce audio-video data and abundant, readily available text-audio datasets.
Methods
MMAudio introduces a streamlined, end-to-end multimodal transformer network based on Flow Matching—an alternative to generative diffusion that models straight-line paths between noise and data for faster inference.
The architecture relies on three key innovations:
- Multimodal Joint Training: Instead of freezing specific modalities, MMAudio maps video, text, and audio into a single transformer network (leveraging an MM-DiT block design). During training, missing modalities are randomly masked. This simple trick allows the network to learn from both tri-modal (video-text-audio) and bi-modal (text-audio) datasets seamlessly, effectively scaling its data exposure.
- Conditional Synchronization Module: Standard cross-attention layers tend to blur temporal details because they aggregate features via a soft distribution. To achieve razor-sharp timing, MMAudio extracts high-frame-rate (24 fps) visual features using a pretrained, self-supervised desynchronization detector (Synchformer). It then injects these features directly into the audio stream via token-level adaptive layer normalization (adaLN) layers.
- Aligned RoPE & ConvMLP: To handle different frame rates (8 fps for standard video vs. 31.25 fps for audio latents), the authors scale the frequencies of the visual Rotary Position Embeddings (RoPE) proportionally. Additionally, they replace standard MLPs in the visual and audio streams with 1D convolutional MLPs (ConvMLPs) to better capture local temporal structures.
Datasets
The model was trained and evaluated on several prominent datasets:
- Audio-Visual-Text: VGGSound (~500 hours of video with class labels used as text prompts).
- Audio-Text: AudioCaps (~128 hours), Clotho (~31 hours), and WavCaps (~7,600 hours), totaling roughly 951K audio-text pairs.
- Evaluation Benchmarks: The VGGSound test set, the AudioCaps test set (to evaluate text-to-audio capabilities), and the Greatest Hits dataset (used specifically for model-free, precise temporal alignment evaluation).
Note on Data Integrity: The authors identified and meticulously removed minor train/test overlaps (<1%) across these popular datasets to ensure completely unbiased evaluations—a contribution highly valuable to the broader research community.
Key Insights and Ideas
- Joint Training is a Win-Win: Exposing the model to pure text-audio data doesn't confuse its video-to-audio capabilities; instead, it provides a massive boost. The shared semantic space helps the model learn a much richer distribution of natural sounds, resulting in a 10% lower Fréchet Distance and 14% better synchronization scores.
- Normalization Injection Beats Soft Attention: For sub-second synchronization, treating visual timing cues as a modulation bias in the normalization layers (adaLN) provides significantly tighter alignment than relying purely on the soft weights of cross-attention mechanisms.
- Asymmetric Layering Saves Compute: By stacking a sequence of multimodal transformer blocks followed by a sequence of audio-only blocks, MMAudio builds a deeper, more expressive network without inflating the parameter count or computational overhead.
Key Results
MMAudio establishes a new state-of-the-art for public video-to-audio models across multiple dimensions:
- Efficiency: The smallest model variant (S-16kHz, 157M parameters) generates high-quality 8-second audio clips in just 1.23 seconds on an NVIDIA H100 GPU, outperforming baselines that are nearly ten times larger.
- Temporal Precision: On the Greatest Hits dataset, which measures the exact onset accuracy of physical impacts (like a drumstick hitting a surface), MMAudio achieved an onset accuracy of 0.7637, vastly outperforming prior models like FoleyCrafter (0.4533).
- Commercial Competitiveness: When benchmarked against Meta's massive, private Movie Gen Audio (a 13B parameter model trained on 100x more data), MMAudio achieved identical subjective audio quality scores and demonstrated superior video-audio synchrony (DeSync score of 0.771 vs. Movie Gen's 1.006).
Limitations and Future Work
The primary limitation acknowledged by the authors is human speech synthesis. When presented with video clips of moving mouths or text prompts for speech, MMAudio generates "unintelligible mumbles." Human speech possesses intricate linguistic, tonal, and grammatical structures that general-purpose Foley models are currently unequipped to handle. The authors point to integrating specialized speech generation modules as a promising direction for future work.
Summary updated