Sequential Contrastive Audio-Visual Learning

Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh, Joan Serrà

Abstract

Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets. However, conventional contrastive audio-visual learning (CAV) methodologies often rely on aggregated representations derived through temporal aggregation, neglecting the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences. In response to this limitation, we propose sequential contrastive audiovisual learning (SCAV), which contrasts examples based on their non-aggregated representation space using multidimensional sequential distances. Audio-visual retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, with up to 3.5x relative improvements in recall against traditional aggregation-based contrastive learning and other previously proposed methods, which utilize more parameters and data. We also show that models trained with SCAV exhibit a significant degree of flexibility regarding the metric employed for retrieval, allowing us to use a hybrid retrieval approach that is both effective and efficient.

Introduction: The Aggregation Bottleneck

Standard audio-visual models often "squash" entire video clips into a single summary vector, losing crucial timing and rhythm information. This process, known as temporal aggregation (e.g., mean pooling), creates a bottleneck that over-compresses data. For instance, two different musical performances might have similar average visual and audio profiles, making them indistinguishable to a model that lacks fine-grained temporal awareness. To tell them apart, a model needs to see the fine-grained timing—the specific moment a hand moves and a note rings out.

Comparison of standard aggregation-based contrastive learning versus the proposed sequential approach.

Methodology: Sequential Contrastive Learning (SCAV)

To address this, we introduce Sequential Contrastive Audio-Visual Learning (SCAV). Instead of comparing single global vectors, SCAV compares full sequences of features, preserving the temporal flow and fine-grained interactions between modalities.

Architecture

The framework utilizes high-quality frozen backbones: CLIP for visual frames and BEATs for audio. Transformer encoders are added on top of these backbones to learn how features relate to each other over time.

Detailed architecture of the SCAV framework, showing the flow from raw inputs to the sequential distance matrix.

Sequential Contrastive Loss

The core of SCAV is the Sequential Contrastive Loss ($L_{seq}$). Rather than simple cosine similarity, the model calculates a distance matrix between sequences. The authors found that an Interpolated Euclidean distance—which involves resizing sequences to match lengths before calculating average squared distance—outperformed more complex alignment algorithms like Dynamic Time Warping (DTW) in terms of scalability and performance.

Normalization and Hybrid Retrieval

A key technical finding was the necessity of "z-score" normalizing the distance matrix (row-wise and column-wise) to magnify differences between correct and incorrect pairs in high-dimensional space. To maintain efficiency during inference, a hybrid retrieval approach is used: a fast global search identifies the top $k$ candidates, which are then re-ranked using the precise sequential metric. This reduces GPU processing time from 13.9s to 0.5s for 1,000 queries while maintaining high accuracy.

Experimental Results

The effectiveness of SCAV was evaluated on the VGGSound and Music datasets. The Music dataset is particularly challenging as it contains semantically similar clips (e.g., different people playing the same instrument), requiring fine-grained temporal discrimination.

  • Massive Gains in Recall: On the VGGSound test set, SCAV achieved a Recall@1 of 22.6%, nearly doubling the performance of the standard aggregation-based (CAV) baseline (12.2%).
  • Superior Zero-Shot Performance: On the challenging Music dataset, SCAV outperformed state-of-the-art models like ImageBind and CAV-MAE by 3 to 3.5x, despite using significantly fewer training parameters and less data.
  • Flexibility: The models proved flexible regarding the metric used for retrieval, allowing researchers to choose between maximum accuracy or maximum speed depending on the application.

Conclusion and Future Work

SCAV demonstrates that temporal information is not just auxiliary but a primary signal for disambiguating complex audio-visual scenes. While the current implementation uses frozen backbones and faces memory constraints due to sequence-to-sequence comparisons, future work involving end-to-end fine-tuning and expansion to other modalities like text holds significant promise for synchronized and rhythmically accurate multimodal generation.

Summary updated

Abstract

Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets. However, conventional contrastive audio-visual learning (CAV) methodologies often rely on aggregated representations derived through temporal aggregation, neglecting the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences. In response to this limitation, we propose sequential contrastive audiovisual learning (SCAV), which contrasts examples based on their non-aggregated representation space using multidimensional sequential distances. Audio-visual retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, with up to 3.5x relative improvements in recall against traditional aggregation-based contrastive learning and other previously proposed methods, which utilize more parameters and data. We also show that models trained with SCAV exhibit a significant degree of flexibility regarding the metric employed for retrieval, allowing us to use a hybrid retrieval approach that is both effective and efficient.
BibTeX
@inproceedings{tsiamas2025sequential,
  author = {Ioannis Tsiamas and Santiago Pascual and Chunghsin Yeh and Joan Serrà},
  title = {{Sequential Contrastive Audio-Visual Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888656},
}
Figure