Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, Longbo Huang

In theory, providing a machine learning model with more data should lead to better performance. If you are training a system to recognize actions in a video, feeding it both the video frames and the audio track should intuitively yield better results than using just one or the other.

However, researchers have frequently observed a frustrating paradox: a model trained on a single modality (like video only) often outperforms a multi-modal model trained on multiple data streams via "late fusion." Until recently, this phenomenon was mostly understood through empirical observation. This paper tackles the fundamental question: What happens during the training process itself that causes multi-modal learning to collapse?

Moving beyond the assumption that "it just happens," the authors provide a provable, mathematical explanation for why naive joint training fails to exploit complementary signals.

The Root Cause: Modality Competition

The core insight of the paper is that joint training acts as a zero-sum game in its early stages. The authors analyze the training dynamics of a late-fusion neural network using a sparse coding model and Gradient Descent (GD). They break the training process into two distinct phases:

  1. Phase 1 (Competition): Near random initialization, the modalities compete. Because the loss function is shared, the network looks for the fastest path to reduce error. The modality that happens to correlate slightly better with the initial random weights grows faster.
  2. Phase 2 (Convergence): Once a "winner" emerges, the overall loss decreases significantly. The network becomes "lazy" and relies entirely on the winning modality. The "losing" modality's encoder gets stuck at its initial level, never learning useful features.

This creates a "rich get richer" scenario. Even if Modality B contains vital complementary information, if Modality A wins the initial lottery of random initialization, Modality B is effectively ignored.

Bar charts demonstrating how one modality often lacks sufficient structure for certain classes, leading to dominance by the other modality.

Provable Sub-optimality and Feature Decay

The authors provide theoretical guarantees proving that while a multi-modal network can achieve zero training error, its test error often remains high. It effectively becomes a uni-modal network in disguise, failing to learn the losing modality's features.

A striking finding is the concept of feature representation decay. If you take a jointly trained multi-modal model and try to use the "losing" modality's encoder for a new task, it performs significantly worse than an encoder trained on that modality alone from scratch. The joint training process actually inhibits feature learning for the loser.

Empirical Evidence

The authors validated their theoretical findings using two primary datasets: Kinetics-400 (Video and Audio) and an Internal Product Dataset (Images and Text).

On Kinetics-400, they demonstrated that for certain classes, the accuracy of a specific modality actually drops when trained jointly compared to when it is trained alone. For example, in classes where audio is structurally weaker or slower to learn, it loses the competition entirely. The network never explores those audio features, resulting in near-zero accuracy for that specific modality stream.

Empirical evidence of modality competition showing error curves and class-level accuracy drops where one modality fails to learn.

Limitations and the Path Forward

While this paper provides a robust theoretical foundation for understanding modality competition, it does have limitations:

  • Simplified Architecture: The mathematical proofs rely on a one-layer encoder and a specific "smoothed ReLU" activation. Real-world models are much deeper and use complex architectures like Transformers.
  • Late Fusion Focus: The study focuses specifically on late fusion (summing features at the end of the network). Other fusion strategies, such as mid-fusion or attention-based fusion, might exhibit different training dynamics.

Despite these limitations, the paper clearly identifies why naive joint training fails. It suggests that future research should focus on adaptive participation or regularization techniques—like gradient modulation—to ensure all modalities are "given a chance" to be learned, rather than being overshadowed by early winners.

Summary updated

Abstract

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the joi...
BibTeX
@inproceedings{huang2022modality,
  author = {Yu Huang and Junyang Lin and Chang Zhou and Hongxia Yang and Longbo Huang},
  title = {{Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=zBr-9gXD4AVy},
}
Figure