Multi-Modal Image Fusion via Intervention-Stable Feature Learning

Xue Wang, Zheng Guan, Wenhua Qian, Chengchao Wang, Runzhuo MaarXiv 2026 · DOI

TL;DR

This paper introduces a novel framework for multi-modal image fusion that moves beyond traditional correlation-based methods. By employing causal intervention strategies, it learns to identify features that are genuinely important for fusion, leading to more robust and generalizable results, especially under challenging conditions.

What Problem Does This Paper Address?

Multi-modal image fusion (MMIF) aims to combine information from different sources (like infrared and visible light images) into a single, more informative image. Existing methods often rely on statistical correlations found in training data. The problem is that these correlations can be spurious, meaning they don't represent true underlying relationships but rather coincidences in the training set. This leads to models that perform well on familiar data but fail when faced with slightly different conditions (out-of-distribution shifts), such as variations in lighting or sensor characteristics. Essentially, these models learn "shortcuts" based on what often appears together, rather than understanding what truly makes the modalities complementary.

Methods

The core of this paper's contribution is an "intervention-based framework" inspired by causal inference principles. Instead of passively observing data, the model actively probes the relationships between modalities through structured perturbations. They design three specific "intervention strategies":

  1. Complementary Masking: Spatially disjoint regions of the input images are masked out. This tests if one modality can compensate for missing information in the other, revealing true complementarity.
  2. Random Masking: Identical regions across modalities are masked. This identifies feature subsets that remain informative even when parts of the input are corrupted, highlighting robust local dependencies.
  3. Modality Dropout: Entire modalities are removed. This quantifies the unique contribution of each modality, preventing the model from becoming overly reliant on one source.

These interventions are used to train a Causal Feature Integrator (CFI) module. The CFI uses "adaptive invariance gating" to learn which features remain important across these different perturbations. This allows it to prioritize stable, genuine cross-modal dependencies over spurious correlations. The overall architecture is a U-Net-like Siamese network with the CFI integrated into the decoder.

Datasets

The paper uses the following datasets:

  • MSRS: Used for training and evaluation in Infrared and Visible Image Fusion (IVIF).
  • RoadScene: Used for validation in IVIF.
  • TNO: A benchmark dataset for IVIF evaluation.
  • M³FD: Used for IVIF evaluation and also for object detection tasks.
  • Harvard Medical Dataset: Used for cross-domain transfer experiments in Medical Image Fusion (MIF).

Key Insights and Ideas

  • Causality over Correlation: The fundamental insight is that true understanding and robust generalization in multi-modal fusion come from identifying causal relationships between modalities, not just statistical correlations.
  • Intervention as a Diagnostic Tool: Actively perturbing the input data (through masking) is a powerful way to diagnose and filter out spurious correlations that plague passive, correlation-based learning.
  • Feature Stability as a Metric: Features that remain consistently important across various "interventions" are likely to be genuinely informative and robust, making them ideal for fusion.
  • CFI for Prioritization: The CFI module acts as an intelligent gatekeeper, learning to weigh features based on their stability under interventions, leading to more interpretable and reliable fusion.
  • Cross-Domain Generalization: The framework's ability to perform well on medical images after being trained on visible-infrared images highlights the generality of the learned "intervention-stable" principles.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art (SOTA) performance on multiple public benchmarks for Infrared and Visible Image Fusion (IVIF), as evidenced by metrics like PSNR, SF, AG, CC, and Qabf.
  • Robustness: The intervention-based approach leads to significantly improved generalization, particularly under challenging conditions where correlation-based methods fail.
  • High-Level Vision Tasks: The fused images produced by the method show superior performance in downstream tasks like object detection (higher mAP) and semantic segmentation (better class-wise performance).
  • Cross-Domain Transfer: The model demonstrates strong performance on medical image fusion (MIF) without any fine-tuning, showcasing its ability to capture fundamental fusion principles applicable across different modalities.
  • Ablation Studies: Removing specific components (like the CFI or intervention losses) leads to noticeable degradation in performance, validating the contribution of each part of the proposed framework.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of multiple interventions during training inherently increases computational overhead compared to standard training.
  • Hyperparameter Tuning: The paper mentions setting hyperparameters like mask sizes and loss weights empirically, suggesting that optimal values might require careful tuning for different datasets or tasks.
  • Future Work: The paper implicitly suggests future work could involve exploring more sophisticated intervention strategies, applying the framework to other multi-modal fusion tasks beyond IVIF and MIF, and further investigating the interpretability of the learned invariance gates. The success in cross-domain transfer also opens avenues for exploring its application in a wider range of multi-modal scenarios.

Summary updated

BibTeX
@misc{wang2026multi,
  author = {Xue Wang and Zheng Guan and Wenhua Qian and Chengchao Wang and Runzhuo Ma},
  title = {{Multi-Modal Image Fusion via Intervention-Stable Feature Learning}},
  year = {2026},
  eprint = {2603.23272},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.23272},
}
Figure