Abstract
It has been crucial to leverage the rich information of multiple modalities in many tasks. Existing works have tried to design multi-modal networks with descent multi-modal fusion modules. Instead, we focus on improving generalization capability of multi-modal networks, especially the fusion module. Viewing the multi-modal data as different projections of information, we first observe that bad projection can cause poor generalization behaviors of multi-modal networks. Then, motivated by well-generalized network's low sensitivity to perturbation, we propose a novel multi-modal training method, multi-head modularization (MHM). We modularize a multi-modal network as a series of uni-modal embedding, multi-modal embedding, and task-specific head modules. Also, for training, we exploit multiple head modules learned with different datasets, swapping each other. From this, we can make the multi-modal embedding module robust to all the heads with different generalization behaviors. In testing phase, we select one of the head modules not to increase the computational cost. Owing to the perturbation of head modules, though including one selected head, the deployed network is more well-generalized compared to the simply end-to-end learned. We verify the effectiveness of MHM on various multi-modal tasks. We use the state-of-the-art methods as baselines, and show notable performance gain for all the baselines.
Introduction
While combining multiple data modalities (e.g., video, audio, and text) theoretically provides more information, multi-modal networks often struggle with generalization. They frequently exhibit a large gap between training and testing performance. Researchers from Qualcomm AI Research argue that this occurs because different modalities act as different "projections" of information; some projections can be noisy or unreliable, leading the model to overfit on training-specific patterns.
Multi-Head Modularization (MHM)
To address this, the authors propose Multi-Head Modularization (MHM). This framework decomposes a standard multi-modal network into three distinct modules:
- Uni-modal Embeddings ($\theta$): Processes each input type independently.
- Multi-modal Embedding ($\phi$): The fusion layer that integrates the modalities.
- Task-specific Heads ($\psi$): The final layers responsible for prediction.
During training, the dataset is split into $K$ folds, and $K$ different classification heads are used. The training follows an iterative, module-wise approach: the fusion module is frozen while a specific head is trained on a data subset, and then that head is frozen while the fusion module is trained on the remaining data. By constantly swapping heads that have seen different data distributions, the fusion module is forced to learn robust, versatile features that work across any head. At inference time, only one head is used, maintaining standard computational efficiency.
Insights: Modality as Projection
The motivation for MHM stems from the observation that different modality configurations (projections) lead to vastly different generalization gaps, even if training errors are identical. Using toy datasets like "Two Moons" and "Swiss Roll," the authors visualized how specific projection lines can cause the generalization gap to fluctuate significantly (e.g., from 5.6% to 17.5%).
MHM effectively acts as a form of structured perturbation. By swapping heads, the network is forced to become less sensitive to weight changes—a property known as low Generalization Sensitivity (GS)—which is a key indicator of a well-generalized model.
Experimental Results
MHM consistently outperformed state-of-the-art (SOTA) baselines across several multi-modal tasks:
- AVE (Audio-Visual Event Detection): Achieved 78.5% accuracy, a 2.9% gain over the baseline.
- THUMOS14 (Action Localization): Improved mean Average Precision (mAP) by 1.8%.
- CMU-MOSEI (Sentiment Analysis): Reached 50.4% accuracy, outperforming the previous SOTA by 1.3%.
- Generalization Gap: In toy experiments, MHM reduced the generalization gap from 13.7% to as low as 0.3%.
Conclusion
Multi-Head Modularization provides a robust training strategy to mitigate overfitting in multi-modal networks. By modularizing the architecture and employing an iterative head-swapping training regime, MHM ensures that fusion layers learn generalized representations. While the method requires defining clear module boundaries and tuning the number of heads ($K$), its success across diverse tasks suggests it is a powerful tool for improving the reliability of multi-modal AI systems.
Summary updated