Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control

Bingliang Li, Fengyu Yang, Yuxin Mao, Qingwen Ye, Hongkai Chen, Yiran Zhong

Abstract

Tri-Ergon is a novel diffusion-based framework designed for high-fidelity, 44.1 kHz stereo video-to-audio (V2A) generation. By integrating multi-modal prompts (text, audio, and video) and professional loudness standards (LUFS), Tri-Ergon enables precise temporal synchronization and stylistic control for video clips up to 60 seconds, bridging the gap between automated generation and professional Foley artistry.

Introduction

While AI video generation has seen rapid advancement, the corresponding audio generation has lagged behind. Current Video-to-Audio (V2A) models frequently suffer from three primary limitations: a lack of user control over specific sound characteristics, poor temporal alignment between visual actions and auditory cues, and low-quality output (often limited to short, mono, low-sample-rate audio). Tri-Ergon addresses these challenges by providing a framework that allows users to precisely "conduct" a soundtrack, ensuring high-fidelity results that meet professional production standards.

Methodology

Tri-Ergon (named after a historic 1919 sound-on-film system) utilizes a Diffusion Transformer (DiT) backbone. Its architecture is defined by three innovative modules that enhance control and synchronization:

The Tri-Ergon framework architecture, highlighting multi-modal embedding (a), LUFS control (b), timing mechanisms (c), and the core Diffusion Transformer (d).
  • Unbalanced Multi-Modal Prompting: Utilizing a LanguageBind encoder, the model processes text, audio, and video simultaneously. The "unbalanced" training allows the model to handle any combination of these inputs—for instance, generating audio from a video alone, or using a text prompt ("loud ambulance siren") and a reference audio clip to define specific styles or off-screen sounds.
  • LUFS Control: The framework integrates Loudness Units relative to Full Scale (LUFS), a professional broadcast standard. By embedding LUFS data into the diffusion process, the model treats volume as an explicit input condition. Users can manually define a loudness curve or utilize a LUFS Prediction Module (based on DINO-V2) to estimate it from visual cues.
  • Variable Length Timing: Specialized timing embeddings allow the model to overcome the typical 10-second limit of V2A models, supporting synchronized audio for videos up to one minute in length.

Datasets and Evaluation

The researchers introduced MM-V2A, a consolidated dataset comprising 505,300 samples from seven diverse sources, including VGGSound, AudioSet, and BBC Sound Effects. This dataset provides the multi-modal labels necessary for training the unbalanced prompting system. For benchmarking, the model was evaluated on the VGGSound test set against state-of-the-art baselines such as SpecVQGAN and Diff-Foley.

Key Results

Tri-Ergon significantly outperforms previous state-of-the-art models across all major metrics:

  • Audio Quality: It achieved a Fréchet Distance (FD) of 113.21, a substantial improvement over Diff-Foley's 242.90.
  • Semantic Alignment: On the KL Divergence metric, Tri-Ergon-L scored 1.82, nearly doubling the performance of earlier models in matching sound to visual categories.
  • Temporal Synchronization: The model achieved the highest AV-Align score (0.231), demonstrating that LUFS integration successfully keeps audio events (like footsteps) in sync with visual triggers.
Visual comparison of audio generation quality. Tri-Ergon's predicted loudness (white line) closely tracks the measured ground truth (yellow line) and visual events, unlike baseline models.

As shown in the qualitative comparison, Tri-Ergon's ability to track loudness curves ensures that transient sounds, such as footsteps, are rendered with the correct intensity and timing, whereas baseline models often produce "blurred" or misaligned auditory results.

Conclusion

Tri-Ergon represents a significant step toward professional-grade AI Foley generation. By treating loudness as a first-class citizen and supporting multi-modal inputs, it offers unprecedented control for creators. While the current Diffusion Transformer backbone is computationally intensive, future work will focus on more efficient generation techniques, such as consistency models, to enable real-time applications in gaming and live editing.

Summary updated

Abstract

Video-to-audio (V2A) generation utilizes visual-only video features to produce realistic sounds that correspond to the scene. However, current V2A models often lack fine-grained control over the generated audio, especially in terms of loudness variation and the incorporation of multi-modal conditions. To overcome these limitations, we introduce Tri-Ergon, a diffusion-based V2A model that incorporates textual, auditory, and pixel-level visual prompts to enable detailed and semantically rich audio synthesis. Additionally, we introduce Loudness Units relative to Full Scale (LUFS) embedding, which allows for precise manual control of the loudness changes over time for individual audio channels, enabling our model to effectively address the intricate correlation of video and audio in real-world Foley workflows. Tri-Ergon is capable of creating 44.1 kHz high-fidelity stereo audio clips of varying lengths up to 60 seconds, which significantly outperforms existing state-of-the-art V2A methods that typically generate mono audio for a fixed duration.
BibTeX
@inproceedings{li2025tri,
  author = {Bingliang Li and Fengyu Yang and Yuxin Mao and Qingwen Ye and Hongkai Chen and Yiran Zhong},
  title = {{Tri-Ergon: Fine-Grained Video-to-Audio Generation with Multi-Modal Conditions and LUFS Control}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32487},
}
Figure