Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment

Tao Jin, Wang Lin, Ye Wang, Linjun Li, Xize Cheng, Zhou Zhao

Abstract

Current multimodal Transformer models often struggle to perfectly synchronize information across different senses, such as matching a specific word to a corresponding facial expression. This paper introduces Multimodal Contextual Contrast (MCC), a scheme that utilizes contrastive learning to force the "results" of attention mechanisms to align across modalities. This approach significantly improves performance in sentiment analysis, speaker trait recognition, and video retrieval.

The Challenge: Misaligned Attentional Results

In multimodal sequential learning, the goal is to understand events across text, video, and audio. The standard approach utilizes the Transformer architecture's "cross-attention" mechanism to correlate these modalities. For instance, when processing the word "rather" in a transcript (the query), the model attends to video and audio (the keys) to find corresponding frames and sounds.

However, a critical flaw exists: while attentional results are expected to be relevant to the query, they are often not aligned with each other. As shown in Figure 1, a text query for "rather" might correctly identify the audio for "rather" but mistakenly focus on visual frames corresponding to the word "shorter." Existing models lack an explicit mechanism to verify if the visual result matches the audio result for the same query, allowing these "capture mistakes" to persist.

Illustration of the misalignment problem in cross-modal attention where a query identifies different temporal segments across modalities.

Methodology: Multimodal Contextual Contrast (MCC)

The authors propose Multimodal Contextual Contrast (MCC) to rectify these misalignments by treating attentional results from different modalities as a group that must maintain consistency.

The Multimodal Contextual Contrast (MCC) framework for aligning attentional results through positive and negative group contrast.

The MCC framework consists of three primary components:

  1. Local Contrastive Alignment: This fine-grained approach considers the visual and audio vectors corresponding to a specific word at time step t as a "positive group." "Negative groups" are generated by swapping in vectors from different time steps. The model is trained to maximize similarity within the positive group while distancing negative groups.
  2. Global Contrastive Alignment: This coarse-grained strategy ensures that truly aligned groups across the entire sequence receive significantly higher scores than unaligned ones.
  3. Stochastic Expectation Approximation (SEA): To handle the exponential growth of modality pairings (e.g., motion, appearance, speech), the authors employ SEA—a kernel approximation trick. This allows for the calculation of alignment scores in linear time, ensuring scalability across complex sensor suites.

Experimental Evaluation

The researchers evaluated MCC across several diverse benchmarks, including CMU-MOSI/MOSEI for sentiment analysis, the POM dataset for speaker trait recognition, and MSR-VTT/LSMDC for video retrieval.

Key Results

MCC consistently outperformed state-of-the-art baselines across all tasks:

  • Sentiment Analysis: On the CMU-MOSI dataset, MCC achieved a binary accuracy of 83.6%, surpassing previous best Transformer-based models.
  • Trait Recognition: On the POM dataset, MCC improved the average multi-class accuracy from 42.0% to 44.3%, a significant jump for this difficult task.
  • Video Retrieval: On the MSR-VTT dataset, MCC reached an R@5 (Recall at 5) score of 56.4% for text-to-video tasks, a 1.2% improvement over existing competitors.

Qualitatively, visualizations demonstrate that MCC can accurately localize phonetic alphabets and mouth shapes for specific words where standard Transformers fail.

Insights and Future Directions

A core insight of this work is that alignment should happen at the level of results, not just inputs. By using hard negative mining via time-shifting, the model becomes highly sensitive to temporal alignment. While the use of SEA proves that contrastive alignment can scale, the authors note that it remains an approximation. Future work may explore non-sequential multimodal tasks and more sophisticated "hard" negative sampling to further enhance robustness.

Summary updated

Abstract

Transformer-based methods have gone mainstream in multimodal sequential learning. The intra and inter modality interactions are captured by the query-key associations of multi-head attention. In this way, the calculated multimodal contexts (attentional results) are expected to be relevant to the query modality. However, in existing literature, the alignment degree between different calculated attentional results of the same query are under-explored. Based on this concern, we propose a new constrained scheme called Multimodal Contextual Contrast (MCC), which could align the multiple attentional results from both local and global perspectives, making the information capture more efficient. Concretely, the calculated attentional results of different modalities are mapped into a common feature space, those attentional vectors with the same query are considered as a positive group and the remaining sets are negative. From local perspective, we sample the negative groups for a positive group by randomly changing the sequential step of one specific context and keeping the other stay the same. From coarse global perspective, we divide all the contextual groups into two sets (i.e., aligned and unaligned), making the total score of aligned group relatively large. We extend the vectorial inner product operation for more input and calculate the aligned score for each multimodal group. Considering that the computational complexity scales exponentially to the number of modalities, we adopt stochastic expectation approximation (SEA) for the real process. The extensive experimental results on several tasks reveal the effectiveness of our contributions.
BibTeX
@inproceedings{jin2024rethinking,
  author = {Tao Jin and Wang Lin and Ye Wang and Linjun Li and Xize Cheng and Zhou Zhao},
  title = {{Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.287},
}
Figure