In machine learning, a foundational assumption is that more data yields better results. When building systems that interact with the real world, this intuition suggests that combining multiple sensory modalities—such as vision, audio, text, and touch—should inherently improve accuracy and robustness. If a video feed is blurry, the audio track should help fill in the blanks.
However, a growing body of research reveals a counterintuitive reality: jointly training multimodal models often causes them to perform worse than their unimodal counterparts. This phenomenon occurs across diverse domains, from deep reinforcement learning agents losing robustness when touch is added to vision (Chen et al. and Yong Lee), to single-cell multi-omics models degrading when biological data streams are combined (Yun et al.).
To build better multimodal architectures, we must understand the underlying mechanics of why this degradation happens.
The Optimization Bottleneck: Modality Competition
The primary reason multimodal models underperform is not a lack of complementary information, but a failure in how networks optimize during training.
In a provable theoretical analysis, Huang et al. demonstrated that a phenomenon called Modality Competition occurs during gradient descent. When a network is trained on multiple modalities simultaneously, the modalities compete for representation based heavily on random initialization.
[Modality A (e.g., Text)] ---> Faster Gradient Descent ---> Dominates Latent Space
|
[Modality B (e.g., Video)] ---> Stalled Optimization ---> Suppressed / Ignored
If one modality (e.g., text) happens to align slightly better with the task at initialization, the network will rapidly minimize its loss by exploiting that single stream. This causes the gradients for the secondary modality (e.g., video) to stall. As a result, the network suppresses potentially useful signals from the secondary modality, collapsing into a sub-optimal solution that fails to utilize the full breadth of the input data.
The Attention and Information Overload Problem
Even when models manage to process multiple streams, integrating them introduces significant structural noise. Rather than acting as a helpful cue, an additional modality can act as a distractor that overwhelms the model's internal representations.
This "less is more" paradox is highly visible in large-scale language and gaming models:
- Attention Clutter: Dikai Liu et al. observe that excessive input data can overwhelm a model's attention mechanism, leading to lower-quality outputs compared to single-modality processing.
- Prompt Cluttering: When trying to improve multimodal sentiment analysis, Wu et al. found that adding auxiliary data like machine-generated captions or synthetic images clutters the input sequence. This clutter impairs the model's ability to map relationships between inputs and outputs during in-context learning.
- Fragile Fusion: In dynamic game-playing environments, Bie et al. show that adding video or audio streams frequently degrades performance because the underlying fusion mechanisms are too fragile to treat secondary modalities as assets rather than distractions.
Alignment Failures and Cross-Modal Hallucinations
When fusion mechanisms fail to properly synthesize data streams, models suffer from modality collapse or cross-modal hallucinations.
For instance, Nadeem et al. show that noisy or misaligned visual data can cause an audio-visual question-answering system to perform worse than a simple audio-only baseline. Because the model cannot resolve conflicts between what it sees and what it hears, the visual stream introduces more confusion than context.
This misalignment manifests vividly as "ghost-seeing" or "ghost-hearing" in Audio-Visual Large Language Models (AV-LLMs). As evaluated by Kim Sung-Bin et al. on the AVHBench benchmark, models frequently hallucinate sounds simply because they see a related object (e.g., hearing a dog bark just because a dog is visible in a silent video). Processing concurrent streams confuses the model's internal representations, making them highly fragile compared to unimodal inputs.
Empirical Evidence from Modern Benchmarks
The failure of joint reasoning is thoroughly documented across several recent diagnostic benchmarks:
| Benchmark | Focus Area | Key Finding |
|---|---|---|
| OmniMMI (Yuxuan Wang et al.) | Streaming Video Contexts | Models perform worse when integrating audio and visual streams than when using vision alone, highlighting real-time fusion difficulties. |
| MCIF (Papi et al.) | Crosslingual Scientific Talks | Integrating visual data with auditory signals paradoxically degrades performance; models treat secondary modalities as noise. |
| JointAVBench (Chao et al.) | Joint Audio-Visual Reasoning | In spatial and emotional tasks, integrated multimodal models underperform unimodal baselines due to conflicting stream distractions. |
| WorldSense (The authors) | Real-world Omnimodal Videos | Many open-source multimodal models perform near random chance when forced to synthesize concurrent audio and visual signals. |
| DeSIQ (The authors) | Social Intelligence Understanding | Adding video features to verbal transcripts decreases performance, showing a failure to synthesize non-verbal cues like gestures. |
Designing Robust Multimodal Architectures
To overcome modality interference, researchers are moving away from naive concatenation or simple attention-based fusion, focusing instead on architectures that explicitly manage stream reliability and specialization.
1. Enforcing Shared Latent Spaces
Instead of letting modalities compete, models can be forced to find common ground. Chen et al. and Yong Lee introduced MuMMI, which uses a contrastive loss to enforce a shared latent space across different sensors. This ensures that even if one sensor becomes highly unreliable or missing, the model can rely on the shared structure to maintain performance.
2. Routing via Mixture-of-Experts (MoE)
Rather than forcing a single neural network layer to process all modalities simultaneously, networks can route data to specialized sub-networks. Yun et al. developed scMoE, which leverages a Sparse Mixture-of-Experts framework. By assigning specific "experts" to handle distinct data modalities, the architecture prevents conflicting signals from disrupting the global optimization path.
3. Dynamic Reliability Scoring
When one modality is corrupted (e.g., a blurry video feed or background noise), the model must have a mechanism to discount that stream. Hong et al. introduced an audio-visual reliability scoring module (AV-RelScore). By dynamically scoring the quality of each input stream, the model can actively ignore misleading information, preventing the negative transfer that typically degrades multimodal performance.
Key Foundational Work
Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning (2021) — contributed 1 concept elements to this direction.
Text Matters but Speech Influences: A Computational Analysis of Syntactic Ambiguity Resolution (2020) — contributed 1 concept elements to this direction.
Crossmodal Language Grounding in an Embodied Neurocognitive Model (2020) — contributed 1 concept elements to this direction.