Introduction
Integrating audio, visual, and textual data effectively remains a significant challenge in multimodal AI. While bi-modal models like CLIP have achieved great success, tri-modal modeling often struggles with the "modality gap"—the fundamental difference between verbal representations (language) and non-verbal sensory inputs (sights and sounds). Furthermore, existing models frequently treat these streams independently, ignoring the inherent temporal synchronization between audio and video.
CoAVT (Correlated Audio-Visual-Text) addresses these issues by drawing inspiration from the "dual coding theory" of human cognition. By separating verbal and non-verbal processing and bridging them with a specialized query mechanism, CoAVT achieves state-of-the-art performance on text-video retrieval and audio-visual classification tasks, even when trained on significantly smaller datasets than its competitors.
Methods and Architecture
The CoAVT architecture is designed to mimic human cognitive subsystems for language and sensory processing. It consists of three primary components:
- Joint Audio-Visual Encoder (Non-Verbal System): Unlike traditional models that use separate encoders, CoAVT employs a joint encoder to process audio and video simultaneously. This design captures the natural synchronization and complementary nature of audiovisual data.
- Text Encoder (Verbal System): A standard BERT-base model is utilized to handle textual inputs.
- Query Encoder (The Bridge): This core innovation bridges the gap between the verbal and non-verbal systems. It uses learnable embeddings (queries) that interact with audiovisual features via cross-attention, distilling complex sensory data into a format compatible with the language system.
Training Objectives
The model is optimized using three distinct losses for each modality pair (Audio-Text, Visual-Text, and AudioVisual-Text):
- Contrastive Loss: Facilitates coarse-grained alignment between clips and text.
- Matching Loss: A binary classification task to predict if a text-modality pair is a match.
- Language Modeling Loss: A generative task where the model predicts text tokens based on audiovisual features, ensuring deep semantic understanding.
Key Results
CoAVT was pre-trained on AudioSet (1.45M pairs) and evaluated on benchmarks including AudioCaps and Clotho. Despite using nearly 20 times less data than models like VALOR (27.5M samples), CoAVT established new state-of-the-art benchmarks.
On the AudioCaps text-video retrieval task, CoAVT achieved a Recall@10 of 57.6% in zero-shot settings, significantly outperforming previous models like AVR (45.2%). In event classification on AudioSet-20K, it reached 45.0 mAP, surpassing competitive models such as MAViL and CAV-MAE. Qualitative analysis further showed that CoAVT provides more specific and accurate descriptions than baseline models, correctly identifying nuanced sounds like "aircraft motors" versus generic "boat motors."
Conclusion
CoAVT demonstrates that a cognition-inspired architecture can lead to superior data efficiency and performance in multimodal tasks. By utilizing a Query Encoder to bridge the modality gap and a Joint Encoder to exploit audiovisual synchronization, the model creates robust representations that outperform much larger-scale systems. Future work will explore scaling this architecture to even larger datasets and applying it to more complex multimodal reasoning tasks.
Summary updated