TL;DR
Current diffusion-based image fusion methods treat source modalities uniformly across time, ignoring that different visual features emerge at distinct stages of denoising. Dig2DIG introduces Diffusion Information Gains (DIG) to dynamically measure how much unrecovered information remains in each modality, provably tightening the model's generalization error bound. Without requiring any fine-tuning or training, Dig2DIG achieves state-of-the-art fusion quality across diverse tasks while cutting diffusion sampling runtime by roughly 70%.
What Problem Does This Paper Address?
Image fusion merges complementary inputs—such as visible and infrared images, or multi-exposure/multi-focus captures—into a single composite image with rich details and salient targets. While diffusion models generate realistic images, existing diffusion fusion methods usually apply static or heuristic weights across all denoising steps.
However, diffusion models exhibit a pronounced spatio-temporal imbalance: coarse structures and thermal hotspots converge during early denoising stages, whereas high-frequency textures (like visible backgrounds) appear much later. Forcing equal or fixed guidance at every step can dilute informative features, bias the fused output toward weak modalities, and waste computational resources.
Methods
Dig2DIG formulates dynamic fusion directly within the reverse sampling process of a standard, pre-trained unconditional diffusion model without additional training:
- Generalization Error Upper Bound: The authors theoretically prove that the generalization error of guided diffusion fusion decomposes into a constant term minus the covariance between modality guidance weights ($w_k$) and the residual, not-yet-fused modality information ($I_{k,t}$). Maximizing this positive covariance provably minimizes the upper bound on generalization error.
-
Diffusion Information Gain (DIG): Because true residual information is latent and unobservable at test time, Dig2DIG uses the single-step reconstruction error of each modality as a computable proxy: $$\text{DIG}_k(t) = \ell(\hat{c}_k^t, c_k)$$ where $\hat{c}_k^t$ is the one-step denoised estimate of noisy input $c_k^t$, and $\ell$ is an image distance metric (such as $\ell_2$).
-
Dynamic Guidance Weighting: The weights for multimodal score guidance are derived per region and timestep using a softmax over the DIG values ($w_k(t) \propto \exp(\text{DIG}_k(t))$). Modalities with higher remaining information gain receive stronger guidance.
- Efficient Sampling Schedule: Instead of calculating DIG at every step, DIG is refreshed every $S = 10$ steps, and sampling is executed in just 25 steps (using larger steps at high noise levels and smaller steps near clean data).
Source Modalities (e.g., IR & Visible)
│
[ Add Noise & Single-Step Denoise ]
│
Compute Residual Proxy (DIG)
│
Softmax Normalization (w_k)
│
▼
Dynamic Multimodal Score Guidance
│
Reverse Diffusion Update (x_{t-1})
Datasets
The authors evaluated the method across three distinct fusion paradigms using existing public benchmarks:
- Visible-Infrared Image Fusion (VIF): LLVIP, M3FD, and MSRS.
- Multi-Focus Image Fusion (MFF): MFFW, plus MFI-WHU (used for ground-truth validation of the residual proxy).
- Multi-Exposure Image Fusion (MEF): MEFB, and a multi-exposure dataset with $K > 2$ exposures (Cai et al., 2018).
No new datasets were introduced.
Key Insights and Ideas
- Theoretical Grounding for Dynamic Fusion: Rather than relying on heuristic rules, the authors demonstrate mathematically that the effectiveness of dynamic weighting depends on positive covariance between fusion weights and residual modality information.
- DIG as an Empirical Mirror of Residual Info: On the ground-truth MFI-WHU benchmark, DIG exhibited a Pearson correlation of $r = 0.9345$ with true ground-truth residual error, validating DIG as a reliable, observable proxy for unobserved residual information.
- Both Spatial and Temporal Adaptivity Matter: Ablation showed that applying DIG across both space (regions) and time (denoising steps) significantly outperformed applying it across time alone, because different image regions within the same timestep require guidance from different modalities.
Key Results
- Visible-Infrared Fusion: On the LLVIP dataset, Dig2DIG attained the top scores across five metrics, achieving an MSE of 1464 (compared to the next best, CCF, at 1658) and an SSIM of 1.23. On M3FD and MSRS, it achieved the highest SSIM (1.41 and 1.42) and lowest MSEs.
- Multi-Focus & Multi-Exposure Fusion: On MFFW, Dig2DIG achieved a Standard Deviation (SD) of 72.95 (surpassing CCF's 69.71) and leading Entropy (7.87) and Spatial Frequency (22.60). On MEFB, it achieved the top SD (75.05) and Average Gradient (8.10).
- Inference Speedup: On M3FD, Dig2DIG completed inference in 52 seconds (819 TFLOPS), compared to 180 seconds (2820 TFLOPS) for DDFM and 633 seconds (8505 TFLOPS) for CCF, delivering a ~70% computational savings over baseline diffusion techniques.
- Downstream Utility: Pedestrian detection evaluations with YOLOv5 on LLVIP confirmed that the enhanced contrast and sharpness translated to superior object detection in low-light environments.
Limitations and Future Work
- Noise Sensitivity in Early Steps: At very high noise levels ($t \approx T$), single-step reconstruction error has high variance, making early DIG estimates noisy and necessitating larger initial step sizes.
- Residual Vanishing in Late Steps: As $t \to 0$, the denoised estimate approaches the clean input, causing residual differences to become very small; updating DIG at every late step degrades performance.
- Diffusion Latency vs. CNNs: Although Dig2DIG is significantly faster than competing diffusion methods (25 vs. 100–300 steps), iterative diffusion inference remains noticeably slower than single-pass CNN or Transformer models, posing a challenge for real-time edge deployment.
Summary updated