Deep Exposure Fusion with Deghosting via Homography Estimation and Attention Learning

Sheng-Yeh Chen, Yung-Yu Chuang

TL;DR

This paper introduces a novel deep learning approach for exposure fusion that tackles the persistent challenges of ghosting artifacts and detail loss in dynamic scenes. By integrating homography estimation for camera motion, an attention mechanism for handling moving objects, and adversarial learning for enhanced realism, their method generates high-quality, ghost-free HDR images from just two differently exposed shots.

What Problem Does This Paper Address?

Modern cameras struggle to capture the full dynamic range of scenes, leading to either blown-out highlights or lost details in shadows. While multi-exposure image fusion (MEF) aims to overcome this by combining multiple shots, it falters when there's camera motion or moving objects within the scene. These movements introduce ghosting artifacts and blur, degrading the quality of the fused image. Existing methods often require static scenes, multiple images, or struggle with significant motion, leaving a gap for robust, two-image fusion solutions that can handle real-world, dynamic scenarios.

Methods

The proposed method is a deep convolutional neural network (CNN) with three main components:

  • Homography Estimation Network (Nh): This network estimates a homography matrix to align the overexposed image (I₂) to the underexposed image (I₁). It uses a 4-point parameterization for more stable optimization and is trained on synthetically generated transformations of aligned images.
  • Fusion Network (Na + Nm): This comprises two modules:
  • Attention Network (Na): Inspired by previous work, this module uses an attention mechanism to highlight well-aligned areas in the warped overexposed image, effectively suppressing misaligned or saturated regions.
  • Merging Network (Nm): This U-Net-like architecture fuses the features from the attention network to produce a visually pleasing image. It employs a perceptual loss based on a VGG-19 network to encourage the generation of images with rich color and structure.
  • Discriminator Network (ND): This component uses adversarial learning (specifically, a conditional Patch-GAN) to further refine the generated image. It learns to distinguish between real reference images and the generated fusion, pushing the generator to produce more realistic results.

The entire pipeline takes an underexposed and an overexposed image as input and outputs a single, well-exposed, ghost-free image.

Datasets

  • SICE Dataset: Used for training and evaluation. This dataset contains multi-exposure stacks of static scenes with high-quality reference images. For training the homography network, synthetic transformations were applied to these aligned images. For testing, the authors used photos taken with a handheld mobile phone using exposure bracketing.

Key Insights and Ideas

  • Two-Image Fusion for Dynamic Scenes: The paper demonstrates that it's possible to achieve robust exposure fusion from just two images, even in the presence of camera and object motion, by intelligently combining alignment and attention mechanisms.
  • Homography for Alignment: Using a dedicated deep network for homography estimation is crucial for compensating camera motion, a common source of artifacts in MEF.
  • Attention for Ghosting Mitigation: The attention mechanism effectively focuses on reliable image regions, helping to suppress artifacts arising from moving objects or misalignments that homography alone might not fully correct.
  • Adversarial Learning for Realism: The integration of a discriminator significantly boosts the visual quality and realism of the fused images, making them more indistinguishable from ground truth.
  • Parameterization Matters: The choice of a 4-point homography parameterization is highlighted as important for stable optimization.

Key Results

  • Improved Alignment: The homography estimation network shows fewer misaligned pixels compared to traditional methods like MTB, as evidenced by XOR difference comparisons. The method achieved an 86.63% success rate on 60 pairs of handheld phone photos.
  • Ghost-Free Fusion: Qualitative comparisons (Fig. 4) show that the proposed method produces significantly sharper and ghost-free images compared to existing methods like Ma et al. [2] and DeepFuse [3], especially for dynamic scenes.
  • Ablation Studies:
    • Removing the homography network (w/o Nh) leads to blurry results due to uncorrected camera motion.
  • Removing the attention mechanism (w/o Na) results in sharper images but still exhibits ghosting artifacts in saturated regions.
  • The full model effectively alleviates ghosting and provides sharper details.
  • Training Efficiency: The entire training process takes approximately 27 hours on a single P100 GPU.

Limitations and Future Work

  • Ground Truth Images: The authors note that they do not have ground truth images for their real-world handheld mobile phone test cases, as these were captured dynamically.
  • Reference Image Dependency: While the method aims to produce high-quality images, the adversarial component relies on the quality of the reference images used during training.
  • Future Work: The paper implicitly suggests that further improvements could be made by exploring more sophisticated attention mechanisms or different adversarial loss functions. The success with two images also opens avenues for exploring even more challenging scenarios with greater motion or fewer input images.

Summary updated

Abstract

Modern cameras have limited dynamic ranges and often produce images with saturated or dark regions using a single exposure. Although the problem could be addressed by taking multiple images with different exposures, exposure fusion methods need to deal with ghosting artifacts and detail loss caused by camera motion or moving objects. This paper proposes a deep network for exposure fusion. For reducing the potential ghosting problem, our network only takes two images, an underexposed image and an overexposed one. Our network integrates together thew homography estimation for compensating camera motion, the attention mechanism for correcting remaining misalignment and moving pixels, and adversarial learning for alleviating other remaining artifacts. Experiments on real-world photos taken using handheld mobile phones show that the proposed method can generate high-quality images with faithful detail and vivid color rendition in both dark and bright areas.
BibTeX
@inproceedings{chen2020deep,
  author = {Sheng-Yeh Chen and Yung-Yu Chuang},
  title = {{Deep Exposure Fusion with Deghosting via Homography Estimation and Attention Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ctwLFauT5N},
}
Figure