Introduction
Self-supervised audiovisual learning typically relies on the assumption that a video clip and its corresponding audio track form a "true" pair, while any other audio is a "false" one. While effective, this approach often fails to distinguish between essential scene attributes and transient linguistic content. This paper introduces dub-augmentation, a novel strategy that leverages multilingual movie dubs to train more robust audiovisual models by treating different language tracks for the same scene as natural counterfactuals.
The Problem of Spurious Correlations
Standard contrastive learning models often over-index on specific spoken words to link audio and video. When a model encounters the same visual scene with different dialogue—a common occurrence in global media—it can become confused by the linguistic "noise."
The goal is to help models ignore these linguistic variations and instead focus on higher-level scene attributes, such as background atmosphere, character emotions, and environmental sounds, which remain consistent regardless of the language spoken.
Methodology: Dub-Augmentation
The researchers utilize a contrastive learning framework enhanced by a massive dataset of dubbed content. The core innovation is the use of "secondary" audio tracks during training to decouple semantic speech from the visual context.
- Cross-Modal Contrastive Loss: The model maximizes similarity between a video embedding and both its primary (e.g., English) and secondary (e.g., Spanish) audio embeddings.
- Within-Modal Consistency: An explicit loss term forces embeddings of different language dubs for the same scene to be similar, emphasizing paralinguistic features like tone and pitch over literal word meanings.
- Architecture: The system employs a Multi-scale Vision Transformer (MViT-S) for video and a corresponding transformer-based architecture for audio.
Datasets and Evaluation
The models were pretrained on a proprietary collection of approximately 59,000 video-hours from 20,000 movies and 33,000 TV episodes, covering seven languages (English, Spanish, French, Japanese, German, Italian, and Korean).
Evaluation was conducted across two major benchmarks:
- HEAR Benchmark: Focused on audio tasks like environmental sound classification (ESC-50) and emotion recognition (CREMA-D).
- LVU (Long-form Video Understanding): Tested video-only capabilities in identifying genres, directors, and character relationships.
- LVU-M: A new synthetic multilingual version of the LVU dataset created using AI-driven speech separation and translation tools to facilitate further research.
Key Results and Insights
The proposed approach achieved state-of-the-art (SOTA) results on several benchmarks, including an 85.60% accuracy on ESC-50. Notably, dub-augmentation improved video-only representations as well, suggesting that a deeper understanding of the "soundscape" helps the visual model extract more meaningful scene features.
While there is a slight performance trade-off in language-specific tasks (like keyword spotting), the resulting representations are significantly more robust for general-purpose scene understanding and paralinguistic analysis.
Conclusion
By treating movie dubs as natural counterfactuals, this research demonstrates that "looking similar but sounding different" is a powerful signal for representation learning. This method effectively de-emphasizes linguistic noise in favor of universal environmental and emotional cues, leading to more versatile and robust audiovisual AI.
Summary updated