FusionDN: A Unified Densely Connected Network for Image Fusion

Han Xu, Jiayi Ma, Zhuliang Le, Junjun Jiang, Xiaojie Guo

Introduction

Image fusion is a critical task in computer vision that involves merging complementary information from multiple source images into a single, comprehensive representation. For instance, combining the thermal signatures from an infrared (IR) sensor with the high-resolution textures of a visible-light camera can provide a superior image for applications like surveillance or autonomous driving. However, the field has traditionally been fragmented, with specialized algorithms designed for specific tasks like multi-exposure fusion or multi-focus fusion.

FusionDN addresses these challenges by introducing a unified, unsupervised deep learning framework. It overcomes three primary hurdles: the lack of ground truth labels for training, the task-specificity of existing models, and the "catastrophic forgetting" that occurs when a neural network is trained on a new task sequentially.

Architecture and Methodology

The FusionDN framework is built upon three core pillars: a densely connected network, a data-driven weight block, and Elastic Weight Consolidation (EWC).

The FusionDN architecture featuring the DenseNet core, the weight-guided loss function, and the data-driven weight block.

Densely Connected Network (DenseNet)

At its heart, FusionDN utilizes a DenseNet architecture. Unlike standard convolutional networks, dense blocks connect each layer to every subsequent layer. This design facilitates robust feature propagation and reuse while significantly reducing the total number of parameters. This efficiency is crucial for a model intended to handle multiple complex fusion tasks within a single set of weights.

Data-Driven Weight Block

Since image fusion often lacks ground truth "perfect" images, FusionDN operates in an unsupervised manner. To guide the learning process, the authors developed a weight block that dynamically determines which source image contains more relevant information at a local level. This block calculates weights ($\omega_1, \omega_2$) by combining two metrics:

  1. No-Reference Image Quality Assessment (NR-IQA): This evaluates technical aspects like sharpness and noise levels.
  2. Entropy: This measures the information density and detail within a patch.

By balancing these metrics, the model can intelligently prioritize, for example, the clear texture of a visible image over a noisy infrared patch, or the thermal signature of a pedestrian over a dark visible background.

Elastic Weight Consolidation (EWC)

To achieve a "universal" fuser, the model must learn multiple tasks sequentially without losing performance on earlier ones. FusionDN employs EWC, a technique from continual learning. When transitioning from Task A (e.g., Vis-IR fusion) to Task B (e.g., multi-exposure fusion), EWC identifies the weights most critical for Task A and penalizes significant changes to them during the training of Task B. This allows the network to accumulate knowledge across diverse fusion scenarios without catastrophic forgetting.

Datasets and Tasks

The versatility of FusionDN was validated across three distinct image fusion domains:

  • Infrared-Visible Fusion: Using the RoadScene dataset, which contains 221 pairs of aligned IR and visible images from real-world driving environments.
  • Multi-exposure Fusion: Utilizing a dataset of 60 image pairs with varying exposure levels (under-exposed and over-exposed).
  • Multi-focus Fusion: Evaluated on the Lytro dataset, where images have different focal planes.
Examples of image patches from the three fusion tasks: (a, d) Visible-Infrared, (b) Multi-focus, and (c) Multi-exposure.

Key Results and Insights

FusionDN demonstrates state-of-the-art (SOTA) performance across all tested categories. In infrared-visible fusion, it excels in preserving both thermal information and fine spatial details, outperforming specialized models like FusionGAN. In multi-exposure tasks, it avoids common artifacts like ghosting, achieving high Visual Information Fidelity (VIF). For multi-focus images, it maintains edge sharpness across the entire field of view.

A key insight from this work is that sequential training with EWC is a highly effective alternative to joint training. It allows for a modular expansion of the model's capabilities without the memory and computational burden of training on all datasets simultaneously. Furthermore, the unsupervised nature of the loss function—driven by the intrinsic quality of the source data—makes the model naturally adaptive to different sensor types and environmental conditions.

Conclusion

FusionDN represents a significant step toward a general-purpose image fusion engine. By integrating dense connectivity with continual learning principles and a sophisticated, data-driven loss mechanism, it provides a single, efficient solution for diverse fusion challenges. Future work may extend this framework to handle more than two source images or adapt it for real-time video fusion applications.

Summary updated

Abstract

In this paper, we present a new unsupervised and unified densely connected network for different types of image fusion tasks, termed as FusionDN. In our method, the densely connected network is trained to generate the fused image conditioned on source images. Meanwhile, a weight block is applied to obtain two data-driven weights as the retention degrees of features in different source images, which are the measurement of the quality and the amount of information in them. Losses of similarities based on these weights are applied for unsupervised learning. In addition, we obtain a single model applicable to multiple fusion tasks by applying elastic weight consolidation to avoid forgetting what has been learned from previous tasks when training multiple tasks sequentially, rather than train individual models for every fusion task or jointly train tasks roughly. Qualitative and quantitative results demonstrate the advantages of FusionDN compared with state-of-the-art methods in different fusion tasks.
BibTeX
@inproceedings{xu2020fusiondn,
  author = {Han Xu and Jiayi Ma and Zhuliang Le and Junjun Jiang and Xiaojie Guo},
  title = {{FusionDN: A Unified Densely Connected Network for Image Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6936},
}
Figure