Interactive Multimodal Learning via Flat Gradient Modification

Qing-Yuan Jiang, Zhouyang Chi, Yang YangIJCAI 2025 · DOI

Introduction

Multimodal learning (MML) integrates information from various sources, such as audio, video, and text, to improve model performance. However, a persistent challenge in this field is modality imbalance, where a model disproportionately relies on a dominant modality, effectively ignoring others. This leads to suboptimal generalization and a failure to capture the full richness of multimodal data.

The Plasticity Trap in Multimodal Learning

Existing approaches to mitigate modality imbalance, such as Multimodal Learning with Alternating Unimodal Adaptation (MLA), employ orthogonal projections to facilitate interaction between modalities. While these methods initially help, they often lead to a significant reduction in plasticity—the model's ability to adapt and learn new features as training progresses. As shown in Figure 1, the rigid constraints of orthogonal methods narrow the available optimization paths, causing the model to become "stuck."

Comparison of plasticity across different multimodal learning paradigms.

Interactive Multimodal Learning via Flat Gradient Modification (IGM)

To overcome the plasticity bottleneck, this paper proposes Interactive Multimodal Learning via Flat Gradient Modification (IGM). The core philosophy of IGM is to transfer information between modalities along "flat" directions of the loss landscape, where the objective function is less sensitive to parameter changes.

Technical Framework

The IGM framework consists of two primary components integrated into an alternating training schedule:

  1. Flat Projection-based Gradient Modification (GM): By performing Singular Value Decomposition (SVD) on the feature covariance matrix, IGM identifies directions in the parameter space that correspond to flat regions of the loss landscape. It constructs a modification matrix ($T$) that scales gradient updates, allowing one modality to influence another without disrupting its own learned representations.
  2. Sharpness-Aware Minimization (SAM): To enhance the effectiveness of the gradient modification, IGM incorporates SAM. SAM explicitly seeks out flat minima during optimization, creating a smoother loss landscape that provides a more stable environment for the interaction mechanism.
The IGM framework: SAM-based optimization and flat projection-based gradient modification.

Experimental Evaluation

The researchers evaluated IGM across five benchmarks, including audio-visual (CREMA-D, Kinetics-Sounds), image-text (Twitter2015, Sarcasm), and tri-modal (NVGesture) datasets.

  • Performance Gains: IGM consistently outperformed state-of-the-art baselines. For instance, on the CREMA-D dataset, IGM achieved an accuracy of 81.05%, a substantial improvement over standard MML (64.52%) and MLA (79.43%).
  • Scalability: The method demonstrated robust performance on the NVGesture dataset with an accuracy of 86.93%, showcasing its ability to handle more than two modalities effectively.
  • Efficiency: Despite the inclusion of SVD and SAM, IGM maintains a competitive training overhead, making it a practical choice for large-scale multimodal tasks.

Conclusion

IGM represents a significant step forward in multimodal optimization. By prioritizing plasticity and leveraging the stability of flat loss landscapes, it enables more effective and flexible interaction between modalities. Future work may explore the application of IGM to heterogeneous architectures and its synergy with various late-fusion strategies.

Summary updated

BibTeX
@inproceedings{jiang2025interactive,
  author = {Qing-Yuan Jiang and Zhouyang Chi and Yang Yang},
  title = {{Interactive Multimodal Learning via Flat Gradient Modification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/611},
}
Figure