Abstract
Data representation learning without labels has attracted increasing attention due to its nature that does not require human annotation. Recently, representation learning has been extended to bimodal data, especially sound and image which are closely related to basic human senses. Existing sound and image representation learning methods necessarily require a large number of sound and image with corresponding pairs. Therefore, it is difficult to ensure the effectiveness of the methods in the weakly paired condition, which lacks paired bimodal data. In fact, according to human cognitive studies, the cognitive functions in the human brain for a certain modality can be enhanced by receiving other modalities, even not directly paired ones. Based on the observation, we propose a new problem to deal with the weakly paired condition: How to boost a certain modal representation even by using other unpaired modal data. To address the issue, we introduce a novel bimodal associative memory (BMA-Memory) with key-value switching. It enables to build sound-image association with small paired bimodal data and to boost the built association with the easily obtainable large amount of unpaired data. Through the proposed associative learning, it is possible to reinforce the representation of a certain modality (e.g., sound) even by using other unpaired modal data (e.g., images).
Introduction
Most state-of-the-art bimodal learning methods, such as those used for audio-visual synchronization, rely on massive datasets of perfectly paired samples. However, collecting large-scale "perfect" pairs is often prohibitively expensive. In contrast, unpaired data—such as random images or ambient audio recordings—is abundant. This paper addresses the weakly paired condition: building robust representations of sound and images when paired data is scarce. Inspired by neurobiology, the authors propose a framework to "boost" modal representations using unpaired data from the opposite modality.
Methodology: Bimodal Associative Memory (BMA-Memory)
The core innovation is the Bimodal Associative Memory (BMA-Memory), which acts as a dynamic bridge between modalities rather than just projecting them into a shared latent space.
Key-Value Switching
The memory consists of two sub-memories for images and sounds. When processing an image, the Image Memory serves as the "Key" to retrieve a corresponding "Value" from the Sound Memory. Conversely, when processing sound, the roles are reversed. This mechanism allows the model to "recall" representations of a missing modality.
Learning Strategies
- Paired Associative Learning: The model is initialized on a small set of paired data using a contrastive loss ($\mathcal{L}_{pair}$), establishing basic cross-modal associations.
- Unpaired Associative Learning: To leverage unpaired data, the model employs Cyclic Memory Addressing. For an unpaired input (e.g., sound), the model queries the memory for a "pseudo-image" feature, which is then used to query back into the sound space. Enforcing consistency through this cycle reinforces associative links without requiring ground-truth pairs.
Experiments and Results
The approach was evaluated on benchmarks including ACIVW (9,000 outdoor sound-image pairs), Kinetics-400 (used as unpaired data), and DCASE-2018 (for zero-shot generalization).
- Recognition Performance: On ACIVW, the method achieved 77.8% Top-1 accuracy for images and 95.6% for sound, significantly outperforming supervised baselines and previous self-supervised methods like HearNet.
- Bimodal Retrieval: In sound-to-image retrieval, BMA-Memory reached a Rank-1 accuracy of 0.522, nearly doubling the performance of prior state-of-the-art methods (0.289).
- Data Efficiency: Remarkably, a model trained on only 20% paired data plus unpaired data outperformed a model trained on 100% paired data without the associative memory.
Discussion and Limitations
A key insight is that adding unpaired data from one modality can improve the representation of the other. However, the authors identified a "sweet spot" regarding the volume of unpaired data.
As shown in the analysis of data volume, while 50k unpaired samples provided a significant boost, increasing the volume to 230k samples led to a slight performance dip. This suggests that extreme imbalances between paired and unpaired data may introduce noise or confusion into the memory. Future work will focus on handling such imbalances to scale the model to even larger, noisier datasets.
Summary updated