Sequential Contrastive Audio-Visual Learning

Ioannis Tsiamas, Santiago Pascual, Chunghsin Yeh, Joan Serrà

TL;DR

Standard audio-visual models often "squash" entire video clips into a single summary vector, losing crucial timing and rhythm information. This paper introduces Sequential Contrastive Audio-Visual Learning (SCAV), a method that preserves the temporal flow by comparing full sequences of features. The result is a massive performance boost—up to 3.5x improvement in retrieval tasks—especially in complex scenarios like distinguishing between different musical performances.

What Problem Does This Paper Address?

In the world of multimodal AI, "Contrastive Learning" is the gold standard. It teaches models to pull "matching" pairs (like a video of a drum and the sound of a drum) closer together in a mathematical space while pushing "mismatched" pairs apart.

However, most current methods use temporal aggregation (like mean pooling). They take a 10-second clip and average all the visual frames and audio samples into one "global embedding." The authors argue this is a "bottleneck" that over-compresses the data. If you have two videos of guitarists playing different songs, their "average" visual and audio profiles might look nearly identical. To tell them apart, a model needs to see the fine-grained timing—the specific moment a hand moves and a note rings out. Standard models are "blind" to this sequence-level detail.

Methods

The authors propose a framework that moves away from comparing single vectors to comparing entire sequences.

  1. Architecture: They use high-quality, frozen "backbones"—CLIP for visual frames and BEATs for audio. On top of these, they add Transformer encoders to learn how features relate to each other over time.
  2. Sequential Contrastive Loss ($L_{seq}$): Instead of using simple cosine similarity between two vectors, they calculate a distance matrix between two sequences.
  3. Distance Functions: They experimented with complex alignment algorithms like Dynamic Time Warping (DTW) and Wasserstein distance, but found that a simpler Interpolated Euclidean distance worked best. This involves "stretching" or "shrinking" one sequence to match the length of the other and then calculating the average squared distance between them.
  4. Normalization Trick: They found that "z-score" normalizing the distance matrix (row-wise and column-wise) was crucial. This magnifies the differences between correct and incorrect pairs in high-dimensional space.
  5. Hybrid Retrieval: Comparing full sequences is computationally expensive. To solve this, they propose a hybrid approach: use fast "global" search to find the top $k$ candidates, then use the precise "sequential" search to pick the final winner.

Datasets

  • VGGSound: A large-scale dataset of 10-second YouTube clips covering a wide variety of audio-visual events.
  • Music Dataset: A more challenging dataset used for "zero-shot" evaluation. It contains many clips with very similar overall semantics (e.g., different people playing the same instrument), making it the perfect test for fine-grained temporal discrimination.

Key Insights and Ideas

  • Aggregation is a Bottleneck: The "aha moment" is that global embeddings are sufficient for static images but fail for dynamic media. Temporal information isn't just "extra" data; it's the primary signal for disambiguating similar scenes.
  • Simplicity Scales: While DTW is a classic for sequence alignment, the authors found it didn't scale well to large batches. Their Interpolated Euclidean approach is $O(n)$ rather than $O(n \cdot m)$, making it much more practical for deep learning.
  • The Power of the Hybrid: The hybrid retrieval method offers a "best of both worlds" scenario. It achieves the high accuracy of sequence matching while being nearly as fast as standard global vector search (reducing GPU time from 13.9s to 0.5s for 1k queries).

Key Results

  • Massive Gains in Recall: On the VGGSound test set, SCAV achieved a Recall@1 of 22.6%, nearly doubling the performance of the standard aggregation-based (CAV) baseline (12.2%).
  • Superior Zero-Shot Performance: On the challenging Music dataset, SCAV outperformed state-of-the-art models like ImageBind and CAV-MAE by 3 to 3.5x, despite using significantly fewer training parameters and less data.
  • Flexibility: The models proved flexible regarding the metric used for retrieval, allowing researchers to choose between maximum accuracy or maximum speed depending on the application.

Limitations and Future Work

  • Memory Footprint: Comparing every point in every sequence across a large batch ($B \times B \times n \times m$) creates a massive memory footprint, which limits how large the training batches can be.
  • Frozen Backbones: The study used frozen CLIP and BEATs models. While effective, the authors suggest that fine-tuning these backbones end-to-end within the SCAV framework could yield even better results.
  • Expanding Modalities: The authors aim to apply this sequential approach to other modalities, such as text, which could lead to more synchronized and rhythmically accurate text-to-video generation.

Summary updated

Abstract

Contrastive learning has emerged as a powerful technique in audio-visual representation learning, leveraging the natural co-occurrence of audio and visual modalities in webscale video datasets. However, conventional contrastive audio-visual learning (CAV) methodologies often rely on aggregated representations derived through temporal aggregation, neglecting the intrinsic sequential nature of the data. This oversight raises concerns regarding the ability of standard approaches to capture and utilize fine-grained information within sequences. In response to this limitation, we propose sequential contrastive audiovisual learning (SCAV), which contrasts examples based on their non-aggregated representation space using multidimensional sequential distances. Audio-visual retrieval experiments with the VGGSound and Music datasets demonstrate the effectiveness of SCAV, with up to 3.5x relative improvements in recall against traditional aggregation-based contrastive learning and other previously proposed methods, which utilize more parameters and data. We also show that models trained with SCAV exhibit a significant degree of flexibility regarding the metric employed for retrieval, allowing us to use a hybrid retrieval approach that is both effective and efficient.
BibTeX
@inproceedings{tsiamas2025sequential,
  author = {Ioannis Tsiamas and Santiago Pascual and Chunghsin Yeh and Joan Serrà},
  title = {{Sequential Contrastive Audio-Visual Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888656},
}

Loading related content...

Figure