Introduction
Multimodal Learning (MML) is predicated on the idea that integrating diverse data sources—such as audio, video, and text—should yield more robust and accurate models than any single modality alone. However, in practice, researchers frequently encounter "modality imbalance." This phenomenon occurs when a model relies excessively on a dominant modality (e.g., video) while neglecting others (e.g., audio), often resulting in a multimodal system that paradoxically performs worse than its unimodal counterparts.
Existing strategies to mitigate this imbalance typically rely on "deferred" rebalancing, where imbalance is measured in one training step and corrected in the next. This paper argues that such reactive measures are insufficient; by the time the correction is applied, the model has already been updated in an imbalanced state. To address this, the authors propose IPRM (Instantaneous Probe-and-Rebalance Multimodal learning), a proactive approach that recalibrates fusion weights within a single training step.
The IPRM Framework
The core innovation of IPRM is its two-pass forward strategy, which allows the model to detect and correct imbalance instantaneously before any gradient updates occur.
1. The Probe Pass (First Forward)
In the initial pass, the model extracts features from all modalities and fuses them using Geodesic Multimodal Mixup (GMM). Unlike standard linear concatenation, GMM maps features onto a hypersphere, which better captures the complex relationships between heterogeneous data types. The model then calculates the Kullback-Leibler (KL) divergence between unimodal and multimodal predictions. This divergence serves as a "strength meter": a high KL divergence identifies a "weak" or "lazy" modality that is not contributing sufficiently to the final prediction.
2. The Rebalance Pass (Second Forward)
Using the strength scores derived from the probe pass, IPRM instantaneously recalibrates the fusion weights ($\lambda$). A second forward pass is then executed under this balanced condition. Only after this second pass is the final loss calculated and the gradients backpropagated to update the network parameters. To maintain stability across training iterations, an Exponential Moving Average (EMA) is used to update the initial weights for the subsequent probe pass.
Experimental Evaluation
The authors evaluated IPRM across five diverse benchmarks, including audio-visual datasets (CREMA-D, KSounds), trimodal gesture and emotion recognition (NVGesture, IEMOCAP), and image-text sarcasm detection.
Key Results
- State-of-the-Art Performance: IPRM consistently outperformed both unimodal baselines and previous rebalancing methods like OGM and MLA. On the CREMA-D dataset, IPRM achieved an accuracy of 84.27%, representing a significant +17.74% improvement over the best unimodal result.
- Scalability to Multiple Modalities: Unlike many existing methods limited to two modalities, IPRM successfully scaled to the trimodal NVGesture and IEMOCAP datasets.
- Superior Representations: t-SNE visualizations demonstrated that IPRM produces tighter, more discriminative clusters for previously "weak" modalities, proving that the method forces the model to learn meaningful features from all inputs.
- Efficiency: Despite the two-pass requirement, IPRM maintains linear algorithmic complexity ($O(N)$). Training time overhead is minimal; for example, on CREMA-D, training took 57 seconds per epoch compared to 55 seconds for a standard multimodal model.
Conclusion and Future Work
IPRM represents a significant shift from reactive to proactive modality rebalancing. By ensuring that every gradient update is performed under a balanced state, it effectively overcomes the "lazy modality" problem. Future research directions include adapting IPRM for streaming data environments where modality relationships may shift over time, and integrating the probe-and-rebalance logic directly into Transformer-based attention mechanisms.
Summary updated