Abstract
Tri-Ergon is a novel diffusion-based framework designed for high-fidelity, 44.1 kHz stereo video-to-audio (V2A) generation. By integrating multi-modal prompts (text, audio, and video) and professional loudness standards (LUFS), Tri-Ergon enables precise temporal synchronization and stylistic control for video clips up to 60 seconds, bridging the gap between automated generation and professional Foley artistry.
Introduction
While AI video generation has seen rapid advancement, the corresponding audio generation has lagged behind. Current Video-to-Audio (V2A) models frequently suffer from three primary limitations: a lack of user control over specific sound characteristics, poor temporal alignment between visual actions and auditory cues, and low-quality output (often limited to short, mono, low-sample-rate audio). Tri-Ergon addresses these challenges by providing a framework that allows users to precisely "conduct" a soundtrack, ensuring high-fidelity results that meet professional production standards.
Methodology
Tri-Ergon (named after a historic 1919 sound-on-film system) utilizes a Diffusion Transformer (DiT) backbone. Its architecture is defined by three innovative modules that enhance control and synchronization:
- Unbalanced Multi-Modal Prompting: Utilizing a LanguageBind encoder, the model processes text, audio, and video simultaneously. The "unbalanced" training allows the model to handle any combination of these inputs—for instance, generating audio from a video alone, or using a text prompt ("loud ambulance siren") and a reference audio clip to define specific styles or off-screen sounds.
- LUFS Control: The framework integrates Loudness Units relative to Full Scale (LUFS), a professional broadcast standard. By embedding LUFS data into the diffusion process, the model treats volume as an explicit input condition. Users can manually define a loudness curve or utilize a LUFS Prediction Module (based on DINO-V2) to estimate it from visual cues.
- Variable Length Timing: Specialized timing embeddings allow the model to overcome the typical 10-second limit of V2A models, supporting synchronized audio for videos up to one minute in length.
Datasets and Evaluation
The researchers introduced MM-V2A, a consolidated dataset comprising 505,300 samples from seven diverse sources, including VGGSound, AudioSet, and BBC Sound Effects. This dataset provides the multi-modal labels necessary for training the unbalanced prompting system. For benchmarking, the model was evaluated on the VGGSound test set against state-of-the-art baselines such as SpecVQGAN and Diff-Foley.
Key Results
Tri-Ergon significantly outperforms previous state-of-the-art models across all major metrics:
- Audio Quality: It achieved a Fréchet Distance (FD) of 113.21, a substantial improvement over Diff-Foley's 242.90.
- Semantic Alignment: On the KL Divergence metric, Tri-Ergon-L scored 1.82, nearly doubling the performance of earlier models in matching sound to visual categories.
- Temporal Synchronization: The model achieved the highest AV-Align score (0.231), demonstrating that LUFS integration successfully keeps audio events (like footsteps) in sync with visual triggers.
As shown in the qualitative comparison, Tri-Ergon's ability to track loudness curves ensures that transient sounds, such as footsteps, are rendered with the correct intensity and timing, whereas baseline models often produce "blurred" or misaligned auditory results.
Conclusion
Tri-Ergon represents a significant step toward professional-grade AI Foley generation. By treating loudness as a first-class citizen and supporting multi-modal inputs, it offers unprecedented control for creators. While the current Diffusion Transformer backbone is computationally intensive, future work will focus on more efficient generation techniques, such as consistency models, to enable real-time applications in gaming and live editing.
Summary updated