In machine learning, conventional wisdom suggests that incorporating more data streams yields better performance. If a model has access to video, audio, and text simultaneously, it should theoretically outperform a model restricted to just one of those channels.
However, in multimodal systems, this assumption frequently breaks down. When a high-quality data stream is naively combined with a noisy, degraded, or irrelevant secondary modality, the system often performs worse than a single-modality baseline. This phenomenon is known as a fusion penalty or modality contamination. For instance, Wang et al. demonstrate in Diversity-oriented Deep Multi-modal Clustering that forcing alignment between clean and noisy modalities dilutes informative signals, causing the network to align its representations with errors rather than ground truth.
Understanding why this interference occurs—and how modern architectures prevent it—is crucial for building robust multi-sensor systems.
How Naive Fusion Propagates Noise
Standard fusion operations like element-wise addition, concatenation, or simple score averaging treat all inputs as equally trustworthy. This democratic approach is precisely why they fail when environmental conditions change.
Consider a semantic segmentation network processing a visible-light (RGB) image and a thermal infrared frame. During the day, both sensors provide clean, complementary signals. But at night, the RGB frame becomes a dark, noisy matrix of uninformative pixels. As noted by Lin et al. in Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation, simple addition or concatenation allows the noisy features from the dark RGB image to directly pollute the clear structural features captured by the thermal sensor, leading to misclassification.
Naive Fusion (Element-wise Addition):
[Clean Thermal Features] + [Noisy/Dark RGB Features] = [Contaminated Features]
(Performance Drops)
This issue is not unique to computer vision. In information retrieval, Wan et al. show in CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval that averaging similarity scores across video, audio, and text channels allows an irrelevant background audio track to introduce "distracted semantics," obscuring highly accurate visual or textual signals. Early fusion at the input level is particularly vulnerable; as Wang et al. observe, corruption in one local feature set can permanently degrade the representation of stable global signals, preventing downstream layers from recovering the lost information.
Dynamic Gating and Routing
To prevent feature contamination, networks must dynamically assess the environmental context and adjust how modalities interact on a frame-by-frame or sample-by-sample basis. Instead of rigid, fixed fusion architectures, modern approaches employ adaptive routers.
For example, Lu et al. developed AFter (AFter: Attention-based Fusion Router for RGBT Tracking), which replaces static fusion with a Hierarchical Attention Network (HAN). The router dynamically selects the optimal interaction strategy—such as processing a modality independently via self-enhancement when its counterpart is severely degraded, or allowing bidirectional fusion only when both signals are clean.
Adaptive Routing (AFter):
/---> [Clean Modality Only] ----> Safe Output
[Sensors] ---> [Router] --|
\---> [Bidirectional Fusion] ---> Fused Output
In pedestrian detection, Li et al. introduced an illumination-aware sub-network (Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection) that explicitly estimates light levels. By treating illumination as a gating weight, the network automatically suppresses the RGB stream at night, relying almost entirely on thermal data, and smoothly transitions back to a balanced fusion during the day.
Asymmetric and Evidence-Aware Fusion
Another powerful mechanism for mitigating interference is breaking the symmetry of the fusion process. Rather than treating modalities as equal partners, asymmetric architectures designate a primary stream and use secondary streams purely for refinement.
Deng et al. leverage this concept in LiCAF: LiDAR-Camera Asymmetric Fusion for Gait Recognition. By structuring the network so that one modality explicitly guides the filtering of the other, they ensure that noise variations do not propagate symmetrically across the network layers.
Alternatively, models can be designed to explicitly quantify their own ignorance. Ma et al. proposed MoNIG (Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions), which uses evidential deep learning to assign a mathematical reliability score to each data source. When a sensor becomes corrupted, its uncertainty score spikes. The network uses these scores to principledly distrust and down-weight the noisy features before they can interact with healthy streams, effectively neutralizing the risk of cross-modal contamination.