In machine learning, the "more is better" heuristic usually holds true: more data, more parameters, and more features typically lead to better performance. However, in multimodal learning—where models integrate diverse data types like video, audio, and text—this intuition often fails. Researchers have repeatedly observed a counterintuitive phenomenon: adding a new data modality can actually decrease a model's predictive accuracy, a problem known as negative transfer or inter-modal interference.
The Fallacy of More Data
The goal of multimodal fusion is to leverage complementary information across different sensors. For instance, in Multimodal Clinical Benchmark for Emergency Care (MC-BEC), Emma Chen and colleagues found that while integrating EHR data with ECG waveforms should theoretically provide a richer picture of patient health, it often introduces noise that obscures critical signals.
This lack of "performance monotonicity"—where accuracy fails to rise as more modalities are added—is a central challenge. As Chen Chen notes in SimMLM, models often fail to prioritize information-rich inputs over information-poor ones, leading to scenarios where a single-modality baseline outperforms a complex fused model.
Why Fusion Fails: Noise and Dominance
The degradation typically stems from three primary mechanisms:
- Modality Poisoning: When one modality is significantly noisier or less reliable than another, a rigid fusion strategy allows the "bad" data to corrupt the "good" data. Xiang He and Dongcheng Zhao describe this as a "poisoning effect" in S-CMRL, where low-quality data degrades the overall system because the model cannot distinguish between reliable and unreliable streams.
- Mathematical Dominance: In a survey on Multi-View Clustering, Guoqing Chao and Shiliang Sun explain that simple concatenation often fails because one view may mathematically dominate others due to its dimensionality or distribution, causing the algorithm to ignore significant portions of the available information.
- Incompatible Distributions: Different modalities (e.g., genomic sequences vs. fMRI scans) often have vastly different distributions. Jin Liu argues in MMKGL that simple fusion methods fail to account for these reliability levels, leading to inter-modal interference where less expressive modalities suppress the signals of stronger ones.
From Naive Concatenation to Adaptive Fusion
To solve this, the research direction has shifted from "early fusion" (simple vector concatenation) to "adaptive" or "divergence-aware" architectures.
Feature Graph Fusion (FGF), introduced by Shenglan Liu 0001 and Muxin Sun, moves away from naive joining by modeling the underlying relationships between modalities using an Extended Jaccard Graph. This allows the model to capture correlations rather than just stacking noise.
Similarly, Junjie Zhou and colleagues proposed DAMM-Diffusion, which employs a "divergence-aware" mechanism. This system intelligently decides whether to combine multiple data sources or stick to a single reliable source based on the consistency of the data. If the modalities diverge too much, the model effectively "shuts off" the noisy stream to prevent accuracy degradation.
Balancing the Modalities
Another frontier involves mitigating dominant modality bias, where a model becomes "lazy" and relies on the easiest signal (often text) while ignoring others (like images). Junehyoung Kwon and the team behind BALGRAD address this by dynamically reweighting training gradients. By projecting gradients to ensure they don't conflict, they force the model to learn from both modalities even when one is initially more informative.
Ultimately, the field is moving toward selective modality fusion. As demonstrated by Wen Yan 0005 in the HyperCombiner network, the most robust models are those that explicitly model decision rules—much like a human expert—to determine which image sequences actually drive a diagnosis and which are merely distracting noise.