Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, Longbo Huang

Abstract

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network often outperforms the jointly trained multi-modal network—a counter-intuitive result given that multiple signals generally provide more information. This work provides a theoretical explanation for this performance gap within the prevalent joint training framework. Based on a simplified data distribution capturing realistic multi-modal properties, we prove that for multi-modal late-fusion networks with (smoothed) ReLU activation, different modalities compete for the gradient. The encoder networks learn only a subset of modalities, a phenomenon we term "modality competition." The losing modalities, which fail to be discovered, are the source of sub-optimality in joint training. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.

Introduction

In theory, multi-modal learning should be a "win-win" scenario. By integrating diverse data sources—such as vision, audio, and text—a model should theoretically possess a more comprehensive understanding of the task than a model restricted to a single source. However, practitioners frequently encounter a frustrating "performance gap": a model trained on a single modality often outperforms a jointly trained multi-modal counterpart.

This paper addresses the fundamental question: Why does joint training fail to capture the best of all worlds? While previous hypotheses suggested differing overfitting rates between modalities, this research investigates the optimization process itself. The authors demonstrate that the failure is rooted in "Modality Competition," where modalities fight for the gradient during training, often resulting in the suppression of useful information.

The Theoretical Framework

To analyze this phenomenon, the researchers utilized a rigorous mathematical framework consisting of three core components:

  1. The Data Model: They employed a sparse coding model to represent multi-modal data. In this setup, each modality contains both "sufficient" data (features that are easy to learn and highly predictive) and "insufficient" data (where the signal is weak or requires integration with other sources).
  2. The Architecture: The study focused on a late-fusion network. Here, each modality is processed by its own encoder (a one-layer neural network with smoothed ReLU activation). The outputs are then fused via a summation operation and passed to a linear classifier.
  3. The Optimization Dynamics: By tracking the dynamics of Gradient Descent (GD), the authors identified two distinct phases of training:
  4. Phase 1 (Competition): Modalities compete to be "discovered" based on how well their features align with the random initial weights of the network.
  5. Phase 2 (Convergence): Once a modality gains an edge, it dominates the gradient. The "winning" modality reduces the loss so effectively that the gradient signal for other modalities vanishes, leaving them stuck at their initialization levels.

Key Insights: The "Winning the Lottery" Effect

The research formalizes the idea that joint training is subject to a "lottery" at initialization. If the initial weights of an image encoder happen to align better with image features than the text encoder aligns with text features, the image modality "wins" the competition for that specific class.

Once a modality begins to minimize the training loss, the network becomes "lazy." Because the loss is decreasing, the gradient updates for the "losing" modalities become negligible. The model essentially decides it has "solved" the problem using the winning modality and stops looking for complementary features in the others. This leads to a sub-optimal state where the model ignores vital information that could have improved generalization.

Empirical Validation

The authors validated their theory using both synthetic data and a large-scale Internal Product Classification Dataset (~600,000 samples) involving product images and titles.

  • Modality Suppression: In specific categories like "fabric" or "business," the uni-modal image model achieved decent accuracy. However, in the jointly trained multi-modal model, the image encoder's accuracy for those same classes dropped to zero. This confirms that the image modality was completely suppressed by the text modality during joint training.
  • Feature Degradation: The quality of feature representations learned during joint training was significantly lower than those learned during uni-modal training. The joint model failed to learn the underlying structure of the "losing" modality because the "winning" modality provided a shortcut to minimizing training error.

Conclusion and Future Directions

The paper provides a provable guarantee (Theorem 5.2) that while joint training can achieve zero training error, it often results in high test error because the encoders fail to learn the full breadth of available features.

While the study focuses on one-layer encoders and late fusion, it provides a foundational understanding of why simple joint training is often insufficient. The findings suggest that future multi-modal research should move away from simple joint optimization and toward algorithms or regularization techniques that actively prevent modality competition, ensuring that the network explores and integrates all available signals rather than taking the path of least resistance.

Summary updated

Abstract

Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network outperforms the jointly trained multi-modal network, which is counter-intuitive since multiple signals generally bring more information. This work provides a theoretical explanation for the emergence of such performance gap in neural networks for the prevalent joint training framework. Based on a simplified data distribution that captures the realistic property of multi-modal data, we prove that for the multi-modal late-fusion network with (smoothed) ReLU activation trained jointly by gradient descent, different modalities will compete with each other. The encoder networks will learn only a subset of modalities. We refer to this phenomenon as modality competition. The losing modalities, which fail to be discovered, are the origins where the sub-optimality of joint training comes from. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.
BibTeX
@inproceedings{huang2022modality,
  author = {Yu Huang and Junyang Lin and Chang Zhou and Hongxia Yang and Longbo Huang},
  title = {{Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=zBr-9gXD4AVy},
}
Figure