Abstract
Multimodal learning aims to establish a unified discrete representation across diverse data types such as audio, video, and text. While Residual Vector Quantization (RVQ) has proven effective for unimodal compression, its application in multimodal contexts often leads to a "precision trap" where fine-grained numerical reconstruction hinders cross-modal alignment. This paper introduces SRCID (Semantic Residual Cross-modal Information Disentanglement), a framework that replaces numerical residuals with semantic residuals. By disentangling modal-general and modal-specific features, SRCID achieves state-of-the-art performance across multiple benchmarks, including cross-modal retrieval and event localization.
Introduction
The quest for a "unified language" in AI involves mapping different modalities into a shared discrete space, typically using Vector Quantization (VQ). However, a significant challenge arises when attempting to increase representation precision. Traditional methods like RVQ focus on minimizing the numerical residual—the difference between the original signal and its quantized version. While this improves reconstruction for a single modality, it often widens the "modality gap," as the model prioritizes idiosyncratic numerical details over shared semantic meaning.
Methodology: From Numerical to Semantic Residuals
The proposed SRCID framework addresses the precision trap by shifting the focus from numerical subtraction to semantic disentanglement.
Architecture Overview
Unlike traditional RVQ, which performs sequential numerical subtractions, SRCID utilizes a two-layer hierarchical architecture designed to extract shared semantic entities.
- First Layer: Separates the input into "modal-general" (shared) features ($g1$) and "modal-specific" (unique) features ($s1$).
- Second Layer (Semantic Residual): Instead of discarding the specific features, the model treats them as a source of further shared information. It extracts a second set of general features ($g2$) from the specific features ($s1$), effectively capturing "semantic residuals" that simple numerical subtraction would miss.
Disentanglement and Alignment
To ensure the purity of these features, SRCID employs the Contrastive Log-ratio Upper Bound (CLUB) to minimize mutual information between general and specific components, preventing information leakage. Simultaneously, Contrastive Predictive Coding (CPC) is used to maximize the alignment between the general features of corresponding audio and video pairs.
Experimental Results
The model was pre-trained on the VGGSound-AVEL 40k dataset and evaluated on benchmarks including AVE (event classification), AVVP (event localization), and zero-shot retrieval on MSCOCO and Clotho.
Performance Benchmarks
SRCID established a new state-of-the-art (SOTA) with an average score of 62.21 across four major audiovisual tasks, outperforming the previous leading model, DCID (59.58). Notably, in the AVVP Video-to-Audio localization task, SRCID achieved 72.1% accuracy, a significant improvement over DCID's 63.4%.
Ablation and Sensitivity Analysis
The effectiveness of the disentanglement modules was verified through ablation studies. Removing the CLUB modules (w/o club1 or club2) resulted in a noticeable drop in average performance, confirming that strict feature separation is vital for a unified representation.
The researchers also observed that performance is sensitive to codebook size, with an optimal balance found around 400-512 entries. Larger codebooks, while offering higher precision, can lead to overfitting or the re-emergence of the modality gap.
Conclusion and Future Work
SRCID demonstrates that semantic residuals are superior to numerical residuals for multimodal alignment. While the second layer significantly enhances fine-grained tasks like localization, there is a slight trade-off in coarse-grained classification performance. Future research will focus on refining the disentanglement process to prevent noise injection in deeper layers and extending the semantic residual approach to a broader range of modalities.
Summary updated