Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

Donghuo Zeng, Kazushi Ikeda

TL;DR

This paper introduces a novel method for learning joint audio-visual representations by combining cross-modal triplet loss with a progressive self-distillation technique. This approach aims to overcome the limitations of relying solely on explicit labels by dynamically generating "soft" alignment labels, leading to improved performance in cross-modal retrieval tasks.

What Problem Does This Paper Address?

Learning to understand the relationship between audio and visual information is crucial for many AI applications, such as video understanding and content retrieval. Traditional methods often use "metric learning" to project audio and visual data into a shared space where similar items are close and dissimilar items are far apart. However, these methods heavily depend on explicit, often costly, annotations (labels). This reliance can lead to suboptimal performance because it might miss complex, latent relationships within the data that aren't captured by simple labels. Existing advanced methods, while improving on basic approaches, still struggle to fully leverage the inherent distributions and potential correlations within the data, treating all positive and negative samples equally.

Methods

The core of this paper's contribution lies in its novel architecture that integrates two key components:

  • Cross-Modal Triplet Loss with Soft Alignments: Instead of relying solely on hard, pre-defined positive and negative pairs based on labels, the model generates "soft-alignment labels." These are probabilistic associations that capture more nuanced relationships between audio and visual samples, going beyond explicit annotations. This is achieved by using a "teacher" network that learns from a subset of annotated data and then generates these soft labels for the rest of the batch.
  • Progressive Self-Distillation: This technique allows the model to learn from itself. Initially, a teacher network is trained on annotated data. This teacher then generates soft labels to guide a student network. As training progresses, the proportion of data used for generating soft labels increases, effectively allowing the student network to become its own teacher over time. This process is controlled by a parameter 'r' that gradually decreases, making the model more self-reliant and efficient.

The overall objective is to minimize a combined loss function that includes:

  • llab: A standard label-based loss.
  • lcross: The cross-modal triplet loss using the generated soft alignments.
  • ldis: A loss to reduce cross-modal dependency of exact audio-visual pairs, preserving original alignment.

Datasets

The authors evaluated their method on two standard audio-visual datasets:

  • AVE (Audio-Visual Events): Contains 1,955 videos with 15 labels.
  • VEGAS: Contains 28,103 videos with 10 labels.

Key Insights and Ideas

  • Beyond Hard Labels: The most significant insight is that relying solely on explicit, hard labels for metric learning can be a bottleneck. Dynamically generating "soft" alignment labels based on data distributions can unlock richer representations.
  • Self-Distillation for Efficiency: Using self-distillation is a clever way to reduce computational costs associated with traditional knowledge distillation, as it avoids the need for a separate, large pre-trained teacher model.
  • Progressive Learning: The "progressive" aspect of self-distillation is key. It allows the model to gradually transition from relying on explicit labels to trusting its own evolving understanding, leading to a more robust and dynamic learning process.
  • Nuanced Relationships: The method effectively captures subtle similarities and differences between audio and visual content that might be missed by simpler label-based approaches.

Key Results

The proposed method achieved state-of-the-art performance on the audio-visual cross-modal retrieval (AV-CMR) task. Specifically, it demonstrated significant improvements over existing methods:

  • AVE Dataset: Achieved a 2.13% increase in mean average precision (MAP).
  • VEGAS Dataset: Achieved a 1.82% increase in MAP.

Ablation studies also highlighted the importance of each component:

  • Progressive self-distillation improved performance over AADML (a strong baseline) in both training and testing.
  • Eliminating cross-modal dependency (ldis) and using the AA proxy for sample representation were crucial for optimal performance.

Limitations and Future Work

  • Limitations: While the paper doesn't explicitly detail limitations, the reliance on pre-extracted audio (VGGish) and visual (Inception V3) features might be a point of future exploration. The "step-wise schedule" for adjusting 'r' in self-distillation is one of several strategies, and others might yield different results.
  • Future Work: The authors suggest extending their approach to other multimodal scenarios and exploring real-world applications, indicating a broad applicability of their learned embedding techniques.

Summary updated

Abstract

Metric learning projects samples into an embedded space, where similarities and dissimilarities are quantified based on their learned representations. However, existing methods often rely on label-guided representation learning, where representations of different modalities, such as audio and visual data, are aligned based on annotated labels. This approach tends to underutilize latent complex features and potential relationships inherent in the distributions of audio and visual data that are not directly tied to the labels, resulting in suboptimal performance in audio-visual embedding learning. To address this issue, we propose a novel architecture that integrates cross-modal triplet loss with progressive self-distillation. Our method enhances representation learning by leveraging inherent distributions and dynamically refining soft audio-visual alignments -- probabilistic alignments between audio and visual data that capture the inherent relationships beyond explicit labels. Specifically, the model distills audio-visual distribution-based knowledge from annotated labels in a subset of each batch. This self-distilled knowledge is used t
BibTeX
@inproceedings{zeng2025metric,
  author = {Donghuo Zeng and Kazushi Ikeda},
  title = {{Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888698},
}
Figure