In theory, a multimodal model should always outperform a unimodal one. If a system can both see a video and hear its audio, it has more information than a system that only does one or the other. However, researchers have repeatedly observed a "multimodal paradox": joint models often perform worse than their single-modality counterparts.
This failure usually stems from modality imbalance. During training, neural networks are "lazy" optimizers; they gravitate toward the easiest path to reduce loss. If one modality (like text) is more discriminative or converges faster than another (like video), the network will over-rely on that dominant signal. Consequently, the "slower" modality's encoder never fully develops, a phenomenon often called modality collapse.
To solve this, a growing body of research proposes a shared strategy: Auxiliary Unimodal Supervision. By forcing the network to solve the task using each modality independently alongside the joint task, researchers ensure that every input stream contributes meaningful features to the final fusion.
The Mechanism of Auxiliary Losses
The core idea is to transform a standard fusion architecture into a multi-task learning problem. Instead of having a single loss function at the very end of the network, researchers attach auxiliary classification heads to the individual modality branches.
In CentralNet, Valentin Vielzeuf and colleagues demonstrate that maintaining a central aggregator while simultaneously optimizing unimodal losses prevents individual data streams from becoming redundant. This regularization ensures that even if the central network finds a shortcut, the modality-specific branches are still held accountable for their own predictive power.
Similarly, the am-LSTM proposed by Weijun Ji for speech recognition uses auxiliary connections to force the network to master individual signals and cross-modal correlations simultaneously. This prevents the "overfitting" that occurs when a model ignores the nuances of lip movements because the audio signal is "good enough" for most training samples.
Preventing Modality Dominance
A major challenge in multimodal learning is that different sensors have different "learning speeds." In MAGNeto, Giang Ngo and Lam Thanh Do argue that independent auxiliary losses prevent faster-learning modalities from overshadowing slower ones. Without this, the gradients from the dominant modality might dominate the weight updates, effectively "turning off" the other sensors before they have a chance to learn.
Wenchao Xu and colleagues take this further with Prototypical Modal Rebalance (PMR). They use class-specific "prototypes" (centroids in the feature space) to provide stable, independent guidance for slow-learning modalities. By calculating a non-parametric loss based on the distance to these prototypes, they prevent the dominant modality from distorting the gradient updates of the weaker ones.
Enforcing Attribution and Alignment
Beyond simple classification losses, some researchers use regularization to explicitly balance how much the model "listens" to each part. Sahiti Yerramilli and the team at CMU introduced Attribution Regularization, which adds a term to the loss function that penalizes the model if it deviates from a target "attribution ratio" between modalities. This forces the AI to distribute its attention more equitably across all inputs.
In the context of sentiment analysis, Wenmeng Yu and the creators of the CH-SIMS dataset argue that using a single "consensus" label for training is insufficient. Because a person might say something neutral but with an angry tone, they provide fine-grained, modality-specific labels. Training a model to predict these individual labels alongside the unified label forces the sub-networks to capture conflicting signals rather than collapsing into a single, potentially inaccurate average.
Robustness to Missing Data
A significant side benefit of auxiliary supervision is inference-time robustness. If a model is trained only on the fused representation, it often fails catastrophically if one sensor (like a camera) fails.
However, by using auxiliary tasks that require individual encoders to perform the primary task independently—as seen in RFNet for brain tumor segmentation—the model becomes naturally resilient. Xin Yu shows that when individual encoders are forced to be "experts" during training, the system can still produce accurate results even when some MRI modalities are missing at runtime. This "graceful degradation" is a direct result of the encoders being trained to be discriminative on their own, rather than just as part of a committee.
Summary of Intuition
The research direction across these 114 papers suggests that fusion is not enough. To build a truly multimodal system, one must:
- Acknowledge Imbalance: Accept that the model will try to ignore difficult modalities.
- Force Independence: Use auxiliary losses or tasks to ensure each modality branch can solve the problem (or a sub-problem) on its own.
- Regularize Attribution: Explicitly penalize the model for relying too heavily on a single "easy" signal.
By treating multimodal learning as a multi-task problem, researchers like Zehui Wu and Jaywon Koo have shown that we can finally move past the "multimodal paradox" and build systems that are genuinely greater than the sum of their parts.