Pyramid Inter-Attention for High Dynamic Range Imaging

Sungil Choi, Jaehoon Cho, Wonil Song, Jihwan Choe, Jisung Yoo, Kwanghoon SohnSensors 2020 · OpenReview

TL;DR

This paper introduces a novel deep learning approach for creating high-dynamic-range (HDR) images from sequences of low-dynamic-range (LDR) images, specifically tackling the challenge of "ghosting" artifacts caused by significant motion. Their method, featuring a Pyramid Inter-Attention Module (PIAM) for precise alignment and a Dual Excitation Block (DEB) for feature refinement, achieves state-of-the-art results in producing ghost-free HDR images.

What Problem Does This Paper Address?

Standard digital cameras struggle to capture scenes with a wide range of lighting conditions, leading to either blown-out highlights or lost shadow details. High-Dynamic-Range (HDR) imaging aims to overcome this by merging multiple LDR images taken at different exposures to create a more visually rich image. However, a major hurdle arises when there's significant motion (either of the camera or objects within the scene) between these LDR captures. This misalignment causes "ghosting" artifacts, where faint outlines of moving objects appear in the final HDR image. Existing methods, whether relying on traditional optical flow or simpler merging techniques, often fail to adequately address severe motion, leading to these undesirable artifacts.

Methods

The core of this work lies in two novel components integrated into a deep learning framework:

  • Pyramid Inter-Attention Module (PIAM): This module is designed to precisely align LDR image features. It leverages a pyramid structure (processing features at different resolutions) and an "inter-attention" mechanism. Unlike self-attention (which finds similarities within a single image), inter-attention finds correspondences between different LDR images. This allows the network to warp features from non-reference images to match a reference image, effectively handling misalignments. The pyramid approach helps manage computational complexity by first finding coarse correspondences and then refining them locally.
  • Dual Excitation Block (DEB): After alignment, the features are fused. The DEB then recalibrates these fused features. It does this by learning spatial and channel-wise "excitations" that highlight informative features and suppress those that might be corrupted by misalignment or saturation. This ensures that the most relevant information is used for the final HDR reconstruction.

The overall framework involves:

  1. Feature Extraction: Extracting initial features from each LDR image.
  2. Alignment (PIAM): Aligning features from non-reference images to a reference image.
  3. Merging (DEB & DRDB): Fusing the aligned features and then refining them using the DEB and Dilated Residual Dense Blocks (DRDBs) to produce the final HDR image.

Datasets

The proposed method was trained and evaluated on:

  • Kalantari's HDR dataset [19]: This dataset was used for both training and testing. It provides ground truth HDR images, which is crucial for supervised training and quantitative evaluation. It consists of 74 training sets and 15 testing sets, each with three LDR images and a ground truth HDR image.
  • Sen's dataset [11] and Tursun's dataset [24]: These datasets were used for qualitative comparisons, as they do not contain ground truth HDR images.

Key Insights and Ideas

  • Attention for Alignment, Not Just Merging: While attention mechanisms have been used to suppress artifacts in HDR imaging, this paper uniquely employs an "inter-attention" mechanism specifically for aligning LDR features before merging. This is a key distinction from prior work that might use attention to filter features during fusion.
  • Pyramid for Efficiency and Accuracy: The use of a pyramid structure in the PIAM is crucial. It allows for efficient computation of global correspondences at lower resolutions and then finer, local correspondences at higher resolutions, balancing performance and accuracy.
  • Dual Excitation for Feature Refinement: The DEB's ability to recalibrate features both spatially and channel-wise is vital. It goes beyond simple feature fusion by actively learning which features to emphasize and which to suppress, leading to cleaner HDR outputs.
  • Unsupervised Alignment: The PIAM is trained using an unsupervised photometric loss, which is a significant advantage as it doesn't require pixel-level ground truth for alignment.

Key Results

  • State-of-the-Art Performance: The proposed method achieved state-of-the-art results on the Kalantari dataset across multiple evaluation metrics, including PSNR-µ, PSNR-M, PSNR-L, and HDR-VDP-2. For instance, it achieved the highest PSNR-µ of 43.212 and HDR-VDP-2 of 62.481.
  • Ghost-Free HDR: Qualitative comparisons demonstrated that the method effectively eliminates ghosting artifacts, even in scenes with severe object and camera motion, producing visually plausible and detailed HDR images.
  • Improved Alignment Accuracy: The PIAM demonstrated superior matching accuracy compared to traditional SIFT-flow and the PWC-Net deep learning method, as measured by SSIM values.
  • Efficiency: While the PIAM adds to the runtime, the overall method is still significantly faster than methods that rely on traditional optical flow, and competitive with other deep learning approaches, especially considering its robust alignment capabilities.

Limitations and Future Work

  • Runtime: While efficient for its capabilities, the proposed method is still slower than some non-alignment-based methods. The authors note that their method is approximately 60 times faster than Kalantari et al.'s method (which uses classical optical flow) but slower than methods that skip alignment entirely.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the PIAM and DEB suggests potential for further exploration in attention-based alignment and feature recalibration for various image processing tasks. The robustness shown on cellphone images also hints at potential for mobile HDR applications.

Summary updated

Abstract

This paper proposes a novel approach to high-dynamic-range (HDR) imaging of dynamic scenes to eliminate ghosting artifacts in HDR images when in the presence of severe misalignment (large object or camera motion) in input low-dynamic-range (LDR) images. Recent non-flow-based methods suffer from ghosting artifacts in the presence of large object motion. Flow-based methods face the same issue since their optical flow algorithms yield huge alignment errors. To eliminate ghosting artifacts, we propose a simple yet effective alignment network for solving the misalignment. The proposed pyramid inter-attention module (PIAM) performs alignment of LDR features by leveraging inter-attention maps. Additionally, to boost the representation of aligned features in the merging process, we propose a dual excitation block (DEB) that recalibrates each feature both spatially and channel-wise. Exhaustive experimental results demonstrate the effectiveness of the proposed PIAM and DEB, achieving state-of-the-art performance in terms of producing ghost-free HDR images.
BibTeX
@inproceedings{choi2020pyramid,
  author = {Sungil Choi and Jaehoon Cho and Wonil Song and Jihwan Choe and Jisung Yoo and Kwanghoon Sohn},
  title = {{Pyramid Inter-Attention for High Dynamic Range Imaging}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=yWH1kFi7KF},
}
Figure