Abstract
Current multimodal Transformer models often struggle to perfectly synchronize information across different senses, such as matching a specific word to a corresponding facial expression. This paper introduces Multimodal Contextual Contrast (MCC), a scheme that utilizes contrastive learning to force the "results" of attention mechanisms to align across modalities. This approach significantly improves performance in sentiment analysis, speaker trait recognition, and video retrieval.
The Challenge: Misaligned Attentional Results
In multimodal sequential learning, the goal is to understand events across text, video, and audio. The standard approach utilizes the Transformer architecture's "cross-attention" mechanism to correlate these modalities. For instance, when processing the word "rather" in a transcript (the query), the model attends to video and audio (the keys) to find corresponding frames and sounds.
However, a critical flaw exists: while attentional results are expected to be relevant to the query, they are often not aligned with each other. As shown in Figure 1, a text query for "rather" might correctly identify the audio for "rather" but mistakenly focus on visual frames corresponding to the word "shorter." Existing models lack an explicit mechanism to verify if the visual result matches the audio result for the same query, allowing these "capture mistakes" to persist.
Methodology: Multimodal Contextual Contrast (MCC)
The authors propose Multimodal Contextual Contrast (MCC) to rectify these misalignments by treating attentional results from different modalities as a group that must maintain consistency.
The MCC framework consists of three primary components:
- Local Contrastive Alignment: This fine-grained approach considers the visual and audio vectors corresponding to a specific word at time step t as a "positive group." "Negative groups" are generated by swapping in vectors from different time steps. The model is trained to maximize similarity within the positive group while distancing negative groups.
- Global Contrastive Alignment: This coarse-grained strategy ensures that truly aligned groups across the entire sequence receive significantly higher scores than unaligned ones.
- Stochastic Expectation Approximation (SEA): To handle the exponential growth of modality pairings (e.g., motion, appearance, speech), the authors employ SEA—a kernel approximation trick. This allows for the calculation of alignment scores in linear time, ensuring scalability across complex sensor suites.
Experimental Evaluation
The researchers evaluated MCC across several diverse benchmarks, including CMU-MOSI/MOSEI for sentiment analysis, the POM dataset for speaker trait recognition, and MSR-VTT/LSMDC for video retrieval.
Key Results
MCC consistently outperformed state-of-the-art baselines across all tasks:
- Sentiment Analysis: On the CMU-MOSI dataset, MCC achieved a binary accuracy of 83.6%, surpassing previous best Transformer-based models.
- Trait Recognition: On the POM dataset, MCC improved the average multi-class accuracy from 42.0% to 44.3%, a significant jump for this difficult task.
- Video Retrieval: On the MSR-VTT dataset, MCC reached an R@5 (Recall at 5) score of 56.4% for text-to-video tasks, a 1.2% improvement over existing competitors.
Qualitatively, visualizations demonstrate that MCC can accurately localize phonetic alphabets and mouth shapes for specific words where standard Transformers fail.
Insights and Future Directions
A core insight of this work is that alignment should happen at the level of results, not just inputs. By using hard negative mining via time-shifting, the model becomes highly sensitive to temporal alignment. While the use of SEA proves that contrastive alignment can scale, the authors note that it remains an approximation. Future work may explore non-sequential multimodal tasks and more sophisticated "hard" negative sampling to further enhance robustness.
Summary updated