EMEF: Ensemble Multi-Exposure Image Fusion

Renshuai Liu, Chengyang Li, Haitao Cao, Yinglin Zheng, Ming Zeng, Xuan ChengAAAI 2023 · DOI

TL;DR

This paper introduces EMEF, a novel ensemble approach for multi-exposure image fusion (MEF) that leverages the strengths of existing methods without requiring ground truth data or designing new fusion strategies. EMEF achieves state-of-the-art performance by training a single "imitator" network to mimic various MEF techniques and then fine-tuning it with an optimization-based approach to find the best fusion for a given image pair.

What Problem Does This Paper Address?

Capturing high dynamic range (HDR) scenes with standard cameras is challenging, often resulting in underexposed (dark) or overexposed (bright) regions where details are lost. Multi-exposure image fusion (MEF) aims to overcome this by combining multiple low dynamic range (LDR) images of the same scene taken at different exposures into a single, well-exposed image that preserves details across the entire dynamic range.

Despite significant progress, current MEF methods face several challenges:

  • Lack of Real Ground Truth: Synthesized ground truth data, often used for training deep learning models, doesn't perfectly represent real-world fusion results.
  • Subjectivity of HDR: What constitutes a "good" fusion is subjective, making it difficult to define a universal objective metric or loss function that accurately evaluates fusion quality.
  • Method-Specific Strengths and Weaknesses: Existing methods, both traditional and deep learning-based, excel in different aspects (e.g., structural similarity, detail preservation, human perception) but none perform optimally in all scenarios. This leads to the need for an "ensemble" of methods.

The paper addresses the problem of creating a robust and high-quality MEF method that can effectively combine the advantages of various existing techniques without relying on artificial ground truth or complex, handcrafted fusion strategies.

Methods

EMEF employs a two-stage approach:

  1. Imitator Network Pre-training:
    • Ensemble of Imitation Targets: The core idea is to train a single generator network (G) to imitate the fusion behavior of multiple pre-selected MEF methods. These methods are treated as "styles."
    • Style-Modulated GAN: A Generative Adversarial Network (GAN) framework is used. The generator takes a pair of over- and under-exposed images and a "style code" as input, producing a fused image.
  2. Supervised Imitation: During pre-training, the generator is trained in a supervised manner to produce fusion results that closely match the outputs of the target MEF methods for the same input images.
  3. Random Soft Label: To mitigate the "domain gap" between style codes used during pre-training (one-hot) and tuning (continuous), a "random soft label" technique is introduced. This involves replacing the 0s and 1s in one-hot labels with small random numbers within specific ranges, improving generalization and reducing artifacts.
  4. Loss Functions: The imitator network is trained using a combination of SSIM loss (to ensure structural similarity with the target fusion) and adversarial loss (to promote realism).
  5. Imitator Network Tuning (Runtime Optimization):
    • Style Code Optimization: For each new input image pair, EMEF tunes the style code to find the optimal fusion result. This is framed as an optimization problem.
  6. MEF-SSIM Loss: The optimization process aims to minimize a loss function based on the MEF-SSIM metric, which measures how well vital information from the input images is preserved in the fused output.
  7. Gradient Descent: A gradient descent algorithm is used to search for the optimal style code that minimizes the MEF-SSIM loss. This allows EMEF to adapt its fusion strategy to the specific characteristics of each input image pair.

Datasets

  • Training: The SICE dataset was used for training the imitator network. For this purpose, only the brightest and darkest images from 356 static scene sequences were selected.
  • Evaluation: The MEFB (Multi-Exposure Image Fusion Benchmark) dataset was used for evaluation. This dataset contains 100 over-exposure and under-exposure image pairs captured under various conditions.

Key Insights and Ideas

  • Leveraging "Wisdom of the Crowd": The paper's central insight is that instead of trying to create a single "perfect" MEF method, one can combine the strengths of multiple imperfect methods. EMEF achieves this by learning to mimic them.
  • Style Transfer for Fusion: The concept of "style" is ingeniously applied to MEF. By controlling a "style code," the network can adopt the fusion characteristics of different underlying methods, effectively creating a flexible fusion engine.
  • Bridging the Domain Gap: The "random soft label" technique is a crucial innovation for making the two-stage process work effectively. It addresses the common problem of domain shift in GANs and improves the robustness of the learned fusion.
  • Unsupervised Inference: While the imitator network is pre-trained in a supervised manner, the actual fusion inference for a new image pair is unsupervised, relying on an image quality metric (MEF-SSIM) for optimization. This bypasses the need for ground truth during inference.
  • Optimization in Style Space: Instead of searching the vast space of possible fused images, EMEF optimizes a low-dimensional style code. This makes the inference process efficient and effective.

Key Results

  • State-of-the-Art Performance: EMEF achieved the best overall rank on the MEFB benchmark, outperforming both individual ensemble members and other competitive MEF methods across a suite of 12 evaluation metrics.
  • Balanced Performance: EMEF demonstrates a more balanced performance across various metrics compared to individual methods, which tend to excel in specific areas but falter in others. This is attributed to its ability to compensate for the weaknesses of its constituent methods.
  • Qualitative Improvements: Visual comparisons show EMEF producing fused images with better detail preservation, contrast, and fewer artifacts than many competing methods, particularly in challenging regions.
  • Efficiency: The paper notes that generating a 512x512 fusion image takes approximately 1.8 minutes, which is competitive for high-quality MEF.

Limitations and Future Work

  • Computational Cost of Pre-training: While inference is efficient, the pre-training of the imitator network can be computationally intensive.
  • Selection of Ensemble Members: The performance of EMEF is dependent on the selection of the MEF methods included in the ensemble. Choosing suboptimal methods might limit the potential of EMEF.
  • Generalizability to Other Tasks: The paper suggests that the ensemble framework and style-modulation approach could be extended to other image generation tasks beyond MEF.
  • Further Exploration of Style Codes: Future work could explore more sophisticated ways to represent and optimize style codes for even finer control over the fusion process.

Summary updated

BibTeX
@inproceedings{liu2023emef,
  author = {Renshuai Liu and Chengyang Li and Haitao Cao and Yinglin Zheng and Ming Zeng and Xuan Cheng},
  title = {{EMEF: Ensemble Multi-Exposure Image Fusion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25259},
}
Figure