Balancing Multimodal Learning via Online Logit Modulation

Daoming Zong, Chaoyue Ding, Baoxiang Li, Jiakui Li, Ken ZhengIJCAI 2024

Introduction: The Paradox of Multimodal Learning

In theory, integrating multiple data streams—such as video, audio, and text—should consistently improve model performance. However, in practice, multimodal models often paradoxically underperform compared to their single-modality counterparts. This phenomenon occurs because one "dominant" modality (e.g., audio in a music video) often converges much faster than others, causing the model to become "lazy" and ignore the remaining signals.

The Problem: Modality Competition and Logit Divergence

When modalities compete during joint training, the "easy" modality dominates the loss function. As shown in the diagnostic plots below, this imbalance is directly visible in the $L_2$ norms of the logit vectors. The dominant modality's logits explode in magnitude, while the underfitted modality remains flat, leading to a performance gap where the combined model may even perform worse than a unimodal one over time.

Diagnostic plots showing modality imbalance in accuracy and logit norms.

Methods: Harmonizing the Learning Process

To address these issues, the authors propose a two-pronged architecture designed to balance optimization across all modalities: the Intermediate Representation Fusion Block (IRFB) and Online Logit Modulation (OLM).

1. Intermediate Representation Fusion Block (IRFB)

Standard fusion methods often mash features together prematurely. IRFB introduces "multimodal bottleneck tokens" that act as a central communication hub. This design decouples the learning of unimodal features from cross-modal interactions, ensuring that each encoder can develop robust representations without being immediately overwhelmed by the fusion process.

2. Online Logit Modulation (OLM)

OLM is a model-agnostic strategy that dynamically monitors the "health" of each modality. By analyzing training and validation losses in real-time, OLM calculates rebalancing factors. If a modality is overfitting or converging too quickly, OLM scales down its logit magnitudes. Conversely, lagging modalities receive a boost. Crucially, OLM only adjusts the magnitude of the raw output scores, not their direction, preserving the learned logic while equalizing the optimization pace.

Overview of the proposed architecture featuring OLM and IRFB.

Key Results and Insights

The researchers evaluated their approach across diverse benchmarks, including action recognition (VGGSound, AVE), emotion recognition (CREMA-D, MER-MULTI), and scene classification (SUNRGBD).

  • State-of-the-Art Performance: On the CREMA-D dataset, the OLM-based approach achieved 62.4% accuracy, significantly outperforming standard concatenation (51.7%) and previous balancing methods like OGM-GE (61.9%).
  • Efficiency: Unlike prior methods that require multiple training passes or complex gradient sampling, OLM is "online." It operates during a single training run with negligible computational overhead.
  • Improved Robustness: In the MER-MULTI challenge, adding OLM to a transformer backbone improved F1-scores by approximately 2% across various modality combinations (Audio+Video+Text).

Conclusion

By treating multimodal learning as a balancing act rather than a simple fusion task, Online Logit Modulation (OLM) and the Inter-Representation Fusion Block (IRFB) provide a robust framework for training complex models. These methods ensure that every modality contributes its fair share to the final decision, leading to more accurate and reliable multimodal systems.

Summary updated

BibTeX
@inproceedings{zong2024balancing,
  author = {Daoming Zong and Chaoyue Ding and Baoxiang Li and Jiakui Li and Ken Zheng},
  title = {{Balancing Multimodal Learning via Online Logit Modulation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure