TL;DR
This paper introduces KDC-MAE, a novel self-supervised learning (SSL) framework that unifies contrastive learning, masked autoencoding, and knowledge distillation. By employing a complementary masking strategy and a symmetric KL divergence loss, KDC-MAE learns more robust and generalizable multimodal representations, outperforming existing methods on various downstream tasks.
What Problem Does This Paper Address?
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations from unlabeled data, mimicking human "self-teaching." While various SSL objectives like pretext tasks, contrastive learning, and masked autoencoding (MAE) have shown success individually, their optimal combination for learning unified multimodal representations remains an open challenge. Existing MAE approaches, while effective for reconstruction, often lack explicit mechanisms to learn fine-grained correspondences between different modalities (e.g., audio and video). This paper aims to bridge this gap by proposing a method that jointly optimizes these SSL objectives to learn a more comprehensive and modular representation.
Methods
The core of KDC-MAE lies in its novel architecture and training strategy:
- KDC-MAE Architecture: The model utilizes a shared-weight encoder with a dual-head design. This allows it to process two masked versions of the same audio-video input simultaneously. A joint decoder then reconstructs the masked patches.
- Complementary Masking Strategy: Instead of random masking, KDC-MAE employs a "complementary masking" strategy. This involves generating two masks ($M_1$ and $M_2$) for the input tokens such that their intersection is empty ($M_1 \cap M_2 = \emptyset$). This forces the model to learn representations that are robust to different subsets of input information, promoting modular correspondence between modalities.
- Knowledge Distillation (KD) via Symmetric KL Divergence: To foster modular correspondence, the paper introduces a knowledge distillation loss ($L_{kd}$) based on the symmetric Kullback-Leibler (KL) divergence. This loss encourages the latent representations of the two complementary masked inputs to be similar. By using KL divergence on probability distributions derived from the embeddings, it effectively aligns the learned representations without relying on the decoder (which is discarded for downstream tasks).
- Combined Loss Function: The total loss is a weighted sum of the reconstruction loss ($L_r$), contrastive loss ($L_c$), and the proposed knowledge distillation loss ($L_{kd}$). The authors experiment with different weighting schemes, including static and dynamic weights.
Datasets
The paper utilizes several datasets for pretraining and downstream task evaluation:
- AudioSet [14]: Used for SSL pretraining and finetuning on audio-visual tasks. A balanced subset, AudioSet 20k, was also used.
- VGGsound [8]: Another dataset for audio-visual SSL pretraining and finetuning.
- Kinetics-400 [24]: Used for video-only human action recognition finetuning.
- ILSVRC2017 [42]: Used for image classification finetuning.
No new benchmarks were introduced; existing datasets were leveraged to evaluate the proposed method.
Key Insights and Ideas
- Complementary masks are key for modularity: The core idea is that by masking complementary sets of tokens, the model is forced to learn representations that capture information from different parts of the input, leading to better understanding of individual modalities and their relationships.
- Symmetric KL divergence for robust KD: Using symmetric KL divergence for knowledge distillation ensures that the learned representations are aligned regardless of which input is considered the "teacher" or "student," making the distillation process more robust.
- Unifying SSL objectives is beneficial: The paper demonstrates that combining contrastive learning, masked autoencoding, and knowledge distillation in a principled way leads to superior performance compared to using individual objectives or simpler combinations.
- Masking strategy matters more than dynamic weights for some tasks: While dynamic weights for loss mixing were explored, the complementary masking strategy showed a more significant impact on performance, especially for multimodal tasks.
- Video modality can lead to overfitting: For tasks heavily reliant on video (like action recognition), complementary masks can sometimes lead to overfitting due to the inherent complexity and potential for uneven sampling of video data.
Key Results
The proposed KDC-MAE method achieved state-of-the-art results on several downstream tasks:
- Classification Accuracy: KDC-MAE consistently outperformed CAV-MAE and MAKD on various finetuning tasks across AudioSet and VGGsound datasets, showing improvements in both audio-visual and modality-specific classifications. For example, on VGGsound AV classification, the proposed method achieved 64.23% accuracy, compared to 62.44% for MAKD.
- Retrieval, Inpainting, and Localization: On the VGG-Sound dataset, KDC-MAE demonstrated competitive performance in retrieval, inpainting, and visual sound source localization tasks, often outperforming baseline methods.
- Ablation Studies: Experiments showed that the complementary masking strategy, combined with knowledge distillation, significantly boosted performance. The optimal value for the KD loss weight ($\lambda_{kd}$) was found to be 10.
Limitations and Future Work
- Overfitting with Video-Centric Tasks: The paper acknowledges that for video-centric downstream tasks, complementary masks can sometimes lead to overfitting, particularly when the annotation is video-based. This suggests that the effectiveness of complementary masking might be modality-dependent.
- Three-Stream Limitation: The study found that using more than three parallel streams for self-distillation can be counterproductive, potentially due to data overexposure and overfitting.
- Future Work: The authors suggest exploring more advanced masking strategies and investigating dynamic weighting schemes for loss mixing. They also propose further research into optimizing the combination of SSL objectives for even better multimodal representation learning. The work opens doors for more extensive studies in this direction.
Summary updated