Abstract
Despite the remarkable success of deep multi-modal learning in practice, it has not been well-explained in theory. Recently, it has been observed that the best uni-modal network often outperforms the jointly trained multi-modal network—a counter-intuitive result given that multiple signals generally provide more information. This work provides a theoretical explanation for this performance gap within the prevalent joint training framework. Based on a simplified data distribution capturing realistic multi-modal properties, we prove that for multi-modal late-fusion networks with (smoothed) ReLU activation, different modalities compete for the gradient. The encoder networks learn only a subset of modalities, a phenomenon we term "modality competition." The losing modalities, which fail to be discovered, are the source of sub-optimality in joint training. Experimentally, we illustrate that modality competition matches the intrinsic behavior of late-fusion joint training.
Introduction
In theory, multi-modal learning should be a "win-win" scenario. By integrating diverse data sources—such as vision, audio, and text—a model should theoretically possess a more comprehensive understanding of the task than a model restricted to a single source. However, practitioners frequently encounter a frustrating "performance gap": a model trained on a single modality often outperforms a jointly trained multi-modal counterpart.
This paper addresses the fundamental question: Why does joint training fail to capture the best of all worlds? While previous hypotheses suggested differing overfitting rates between modalities, this research investigates the optimization process itself. The authors demonstrate that the failure is rooted in "Modality Competition," where modalities fight for the gradient during training, often resulting in the suppression of useful information.
The Theoretical Framework
To analyze this phenomenon, the researchers utilized a rigorous mathematical framework consisting of three core components:
- The Data Model: They employed a sparse coding model to represent multi-modal data. In this setup, each modality contains both "sufficient" data (features that are easy to learn and highly predictive) and "insufficient" data (where the signal is weak or requires integration with other sources).
- The Architecture: The study focused on a late-fusion network. Here, each modality is processed by its own encoder (a one-layer neural network with smoothed ReLU activation). The outputs are then fused via a summation operation and passed to a linear classifier.
- The Optimization Dynamics: By tracking the dynamics of Gradient Descent (GD), the authors identified two distinct phases of training:
- Phase 1 (Competition): Modalities compete to be "discovered" based on how well their features align with the random initial weights of the network.
- Phase 2 (Convergence): Once a modality gains an edge, it dominates the gradient. The "winning" modality reduces the loss so effectively that the gradient signal for other modalities vanishes, leaving them stuck at their initialization levels.
Key Insights: The "Winning the Lottery" Effect
The research formalizes the idea that joint training is subject to a "lottery" at initialization. If the initial weights of an image encoder happen to align better with image features than the text encoder aligns with text features, the image modality "wins" the competition for that specific class.
Once a modality begins to minimize the training loss, the network becomes "lazy." Because the loss is decreasing, the gradient updates for the "losing" modalities become negligible. The model essentially decides it has "solved" the problem using the winning modality and stops looking for complementary features in the others. This leads to a sub-optimal state where the model ignores vital information that could have improved generalization.
Empirical Validation
The authors validated their theory using both synthetic data and a large-scale Internal Product Classification Dataset (~600,000 samples) involving product images and titles.
- Modality Suppression: In specific categories like "fabric" or "business," the uni-modal image model achieved decent accuracy. However, in the jointly trained multi-modal model, the image encoder's accuracy for those same classes dropped to zero. This confirms that the image modality was completely suppressed by the text modality during joint training.
- Feature Degradation: The quality of feature representations learned during joint training was significantly lower than those learned during uni-modal training. The joint model failed to learn the underlying structure of the "losing" modality because the "winning" modality provided a shortcut to minimizing training error.
Conclusion and Future Directions
The paper provides a provable guarantee (Theorem 5.2) that while joint training can achieve zero training error, it often results in high test error because the encoders fail to learn the full breadth of available features.
While the study focuses on one-layer encoders and late fusion, it provides a foundational understanding of why simple joint training is often insufficient. The findings suggest that future multi-modal research should move away from simple joint optimization and toward algorithms or regularization techniques that actively prevent modality competition, ensuring that the network explores and integrates all available signals rather than taking the path of least resistance.
Summary updated