Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

Nikhil Singh, Chih-Wei Wu, Iroro Orife, Mahdi M. Kalayeh

Introduction

Self-supervised audiovisual learning typically relies on the assumption that a video clip and its corresponding audio track form a "true" pair, while any other audio is a "false" one. While effective, this approach often fails to distinguish between essential scene attributes and transient linguistic content. This paper introduces dub-augmentation, a novel strategy that leverages multilingual movie dubs to train more robust audiovisual models by treating different language tracks for the same scene as natural counterfactuals.

The Problem of Spurious Correlations

Standard contrastive learning models often over-index on specific spoken words to link audio and video. When a model encounters the same visual scene with different dialogue—a common occurrence in global media—it can become confused by the linguistic "noise."

Example of counterfactual pairs in movie dubbing. The visual scene remains identical while the spoken dialogue changes across different language tracks.

The goal is to help models ignore these linguistic variations and instead focus on higher-level scene attributes, such as background atmosphere, character emotions, and environmental sounds, which remain consistent regardless of the language spoken.

Methodology: Dub-Augmentation

The researchers utilize a contrastive learning framework enhanced by a massive dataset of dubbed content. The core innovation is the use of "secondary" audio tracks during training to decouple semantic speech from the visual context.

Overview of the dub-augmentation framework. The model learns to associate a single visual scene with multiple language dubs, focusing on shared environmental sounds (similarities) while ignoring linguistic differences.
  1. Cross-Modal Contrastive Loss: The model maximizes similarity between a video embedding and both its primary (e.g., English) and secondary (e.g., Spanish) audio embeddings.
  2. Within-Modal Consistency: An explicit loss term forces embeddings of different language dubs for the same scene to be similar, emphasizing paralinguistic features like tone and pitch over literal word meanings.
  3. Architecture: The system employs a Multi-scale Vision Transformer (MViT-S) for video and a corresponding transformer-based architecture for audio.

Datasets and Evaluation

The models were pretrained on a proprietary collection of approximately 59,000 video-hours from 20,000 movies and 33,000 TV episodes, covering seven languages (English, Spanish, French, Japanese, German, Italian, and Korean).

Evaluation was conducted across two major benchmarks:

  • HEAR Benchmark: Focused on audio tasks like environmental sound classification (ESC-50) and emotion recognition (CREMA-D).
  • LVU (Long-form Video Understanding): Tested video-only capabilities in identifying genres, directors, and character relationships.
  • LVU-M: A new synthetic multilingual version of the LVU dataset created using AI-driven speech separation and translation tools to facilitate further research.

Key Results and Insights

The proposed approach achieved state-of-the-art (SOTA) results on several benchmarks, including an 85.60% accuracy on ESC-50. Notably, dub-augmentation improved video-only representations as well, suggesting that a deeper understanding of the "soundscape" helps the visual model extract more meaningful scene features.

While there is a slight performance trade-off in language-specific tasks (like keyword spotting), the resulting representations are significantly more robust for general-purpose scene understanding and paralinguistic analysis.

Conclusion

By treating movie dubs as natural counterfactuals, this research demonstrates that "looking similar but sounding different" is a powerful signal for representation learning. This method effectively de-emphasizes linguistic noise in favor of universal environmental and emotional cues, leading to more versatile and robust audiovisual AI.

Summary updated

Abstract

Audiovisual representation learning typically relies on the correspondence between sight and sound. However, there are often multiple audio tracks that can correspond with a visual scene. Consider, for example, different conversations on the same crowded street. The effect of such counterfactual pairs on audiovisual representation learning has not been previously explored. To investigate this, we use dubbed versions of movies and television shows to augment cross-modal contrastive learning. Our approach learns to represent alternate audio tracks, differing only in speech, similarly to the same video. Our results, from a comprehensive set of experiments investigating different training strategies, show this general approach improves performance on a range of downstream auditory and audiovisual tasks, without majorly affecting linguistic task performance overall. These findings highlight the importance of considering speech variation when learning scene-level audiovisual correspondences and suggest that dubbed audio can be a useful augmentation technique for training audiovisual models toward more robust performance on diverse downstream tasks.
BibTeX
@inproceedings{singh2023looking,
  author = {Nikhil Singh and Chih-Wei Wu and Iroro Orife and Mahdi M. Kalayeh},
  title = {{Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52733.2024.02541},
}
Figure