Unsupervised Multi-Exposure Image Fusion Breaking Exposure Limits via Contrastive Learning

Han Xu, Liang Haochen, Jiayi MaAAAI 2023 · OpenReview

Abstract

This paper proposes an unsupervised multi-exposure image fusion (MEF) method via contrastive learning, termed as MEF-CL. It breaks exposure limits and performance bottlenecks faced by existing methods. MEF-CL designs similarity constraints to preserve contents in source images, eliminating the need for artificial ground truth and avoiding the negative impacts of inappropriate training targets. By exploring a latent feature space and applying contrastive learning, the model guides fused images to approximate normal-light samples while staying away from inappropriately exposed ones. Consequently, MEF-CL is applicable to image pairs of any multiple exposures rather than just complementary pairs. Qualitative and quantitative experiments validate its superiority in illumination, texture detail, and color saturation.

Introduction: The Ground Truth Trap and Exposure Limits

Digital cameras often fail to capture the full dynamic range of a scene in a single shot. Multi-exposure image fusion (MEF) addresses this by merging multiple images taken at different exposure levels. However, existing deep learning methods face two major hurdles:

  1. The Ground Truth Trap: Supervised methods require "ground truth" images that do not exist in nature. These are usually created artificially (pseudo-GT), leading to models that inherit the biases and artifacts of the creation process.
  2. Exposure Limits: Most unsupervised methods rely on pixel-level similarity to source images, effectively acting as "interpolators." This forces the fused result to stay between the exposures of the source images, typically requiring a perfectly complementary pair (one under-exposed and one over-exposed). If both source images are under-exposed, the result remains dark.

Methodology: The MEF-CL Framework

The authors propose MEF-CL, an end-to-end unsupervised network consisting of two main components: a Fusion Network and a Contrastive Learning Block.

The MEF-CL architecture featuring a DenseNet fusion network and a contrastive learning block that pulls fused images toward normal-light distributions.

Fusion Network

The core is a 10-layer fully connected DenseNet architecture. It takes two source images as input and produces a fused image. DenseNet was selected for its high feature utilization and ability to prevent information loss during deep processing.

Contrastive Learning Block

This is the framework's "secret sauce." Using a pre-trained VGG-16 network, the model extracts features from the fused image (the anchor), "positive" samples (naturally captured normal-light images), and "negative" samples (the improperly exposed source images). In the latent feature space, the loss function ($L_{InfoNCE}$) "pulls" the fused image closer to the normal-light distribution and "pushes" it away from the under/over-exposed source distributions.

To ensure the result remains true to the scene, the authors combine this contrastive loss with Mean Square Error (MSE) to maintain intensity distribution and Structural Similarity (SSIM) to preserve textures and structural details from the source images.

Breaking the Exposure Limit

Unlike traditional MEF models that find a middle ground, MEF-CL acts as an "extrapolator." Because it understands the characteristics of a high-quality image via contrastive learning, it can take two dark images and produce a well-lit result by moving toward the "normal-light" cluster in feature space.

Visual comparison demonstrating MEF-CL's ability to handle non-complementary exposures (e.g., two under-exposed images) where SOTA methods like U2Fusion fail.

As shown in the comparative results, MEF-CL successfully enhances illumination even when both source images are under-exposed, a scenario where existing methods like U2Fusion typically produce dark, unusable results. Similarly, it can correct over-exposed pairs by pulling them toward the normal-light distribution.

Experimental Results

MEF-CL was evaluated using the SICE dataset and compared against nine state-of-the-art (SOTA) methods, including traditional algorithms (GFF) and deep learning models (U2Fusion, MEF-GAN, TransMEF).

  • Quantitative Performance: MEF-CL achieved top scores in MEF-SSIM (structural similarity), CC (correlation coefficient), and DeltaE (color accuracy).
  • Qualitative Quality: Visual comparisons show that MEF-CL produces images with more saturated colors and fewer artifacts (like halos or ghosting) compared to supervised methods held back by pseudo-GT training data.
  • Efficiency: The model is highly efficient, processing an image pair in approximately 0.033 seconds, making it suitable for real-time applications.

Conclusion

By alleviating the dependence on source image exposure levels and artificial ground truth, MEF-CL shows superior generalization across various scenes. The use of contrastive learning in a latent feature space allows the model to transcend the "interpolation" bottleneck of previous unsupervised methods, delivering images with appropriate illumination, detailed textures, and vivid colors regardless of the input exposure levels.

Summary updated

Abstract

This paper proposes an unsupervised multi-exposure image fusion (MEF) method via contrastive learning, termed as MEF-CL. It breaks exposure limits and performance bottleneck faced by existing methods. MEF-CL firstly designs similarity constraints to preserve contents in source images. It eliminates the need for ground truth (actually not exist and created artificially) and thus avoids negative impacts of inappropriate ground truth on performance and generalization. Moreover, we explore a latent feature space and apply contrastive learning in this space to guide fused image to approximate normal-light samples and stay away from inappropriately exposed ones. In this way, characteristics of fused images (e.g., illumination, colors) can be further improved without being subject to source images. Therefore, MEF-CL is applicable to image pairs of any multiple exposures rather than a pair of under-exposed and over-exposed images mandated by existing methods. By alleviating dependence on source images, MEF-CL shows better generalization for various scenes. Consequently, our results exhibit appropriate illumination, detailed textures, and saturated colors. Qualitative, quantitative, and ablation experiments validate the superiority and generalization of MEF-CL. Our code is publicly available at https://github.com/hanna-xu/MEF-CL.
BibTeX
@inproceedings{xu2023unsupervised,
  author = {Han Xu and Liang Haochen and Jiayi Ma},
  title = {{Unsupervised Multi-Exposure Image Fusion Breaking Exposure Limits via Contrastive Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=yIxZdLcxWm},
}
Figure