Color Event Enhanced Single-Exposure HDR Imaging

Mengyao Cui, Zhigang Wang, Dong Wang, Bin Zhao, Xuelong Li

TL;DR

This paper introduces a novel approach to single-exposure High Dynamic Range (HDR) imaging by incorporating color event data. By fusing information from a single LDR image and asynchronous color events, the method significantly improves the restoration of details and colors in over- and under-exposed regions, overcoming limitations of existing single-image HDR techniques.

What Problem Does This Paper Address?

Capturing scenes with a wide range of lighting conditions (high dynamic range) is a challenge for standard cameras, which produce Low Dynamic Range (LDR) images. These LDR images often lose detail in very bright or very dark areas. While multi-exposure HDR imaging can reconstruct these details, it's computationally expensive and struggles with motion. Single-exposure HDR imaging aims to achieve this with just one LDR image, but it faces a fundamental information loss problem, especially for color restoration in saturated or underexposed regions. Existing methods using monochrome event cameras can help with structural information but don't solve the color distortion issue.

Methods

The core of this work is a joint framework that fuses information from a single LDR image and color event data.

  • Color Event Data: The authors leverage color event cameras, which capture asynchronous pixel-wise color changes in high dynamic range. This data provides edge-like information about color shifts.
  • Exposure-Aware Transformer (EaT): A key innovation is the Exposure-Aware Transformer module. This module is designed to propagate color hints from well-exposed regions and event streams to the missing areas in over-/under-exposed regions. It uses an "exposure-aware mask" to suppress distracting information and focus on relevant color propagation.
  • Image-Event Fusion Blocks: Multi-scale features from both the LDR image and color events are extracted and then fused using specialized blocks that merge color event information with LDR image features at different scales.
  • Region-Based Discriminator: To further enhance the generation of severely missing regions, a region-based discriminator is employed.

Datasets

The authors introduce two types of datasets:

  • Synthesized Datasets: They leverage existing HDR-SYNTH and HDR-REAL datasets and simulate color event data for them using the ESIM simulator.
  • Real-World Dataset (HDR-CE): A new dataset, named HDR Color Event (HDR-CE), is collected using a DAVIS346-color event camera. This dataset consists of synchronized LDR images and corresponding color events.

Key Insights and Ideas

  • Color Events are Crucial for Color HDR: The paper highlights that while monochrome events can help with structure, color events are essential for restoring accurate colors in challenging lighting.
  • Transformer for Cross-Modal Propagation: The Exposure-Aware Transformer is a novel way to leverage the long-range dependency modeling of transformers for propagating color information across different modalities (LDR image and color events) and regions with varying exposure.
  • Exposure-Aware Masking is Key: The exposure-aware mask within the transformer is critical for filtering out noise and ensuring that only relevant color hints are propagated, preventing color distortion.
  • Joint Fusion is More Than Sum of Parts: Simply concatenating features isn't enough. The proposed fusion blocks and the transformer architecture are designed to effectively integrate the complementary information from LDR images and color events.

Key Results

  • State-of-the-Art Performance: The proposed method achieves superior HDR reconstruction quality, outperforming existing methods by significant margins (e.g., 0.47 on HDR-SYNTH and 0.11 on HDR-REAL datasets in terms of HDR-VDP-3 scores).
  • Improved Color and Detail Restoration: Qualitative results demonstrate a significant improvement in restoring details and colors in over- and under-exposed regions compared to baseline methods.
  • Effectiveness of Components: Ablation studies confirm the effectiveness of each proposed component, including color events, the Exposure-Aware Transformer, and the region-based discriminator.

Limitations and Future Work

  • Computational Cost: While single-exposure HDR is more efficient than multi-exposure, the proposed method still involves complex modules like transformers, which can be computationally intensive.
  • Dataset Size: While a new real-world dataset is introduced, the authors might suggest expanding it further for more robust training and evaluation.
  • Future Directions: The paper implicitly suggests future work could explore more efficient transformer architectures, investigate other types of event sensors, or extend the method to video HDR reconstruction.

Summary updated

Abstract

Single-exposure high dynamic range (HDR) imaging aims to reconstruct the wide-range intensities of a scene by using its single low dynamic range (LDR) image, thus providing significant efficiency. Existing methods pay high attention to restoring the luminance by inversing the tone-mapping process, while the color in the over-/under-exposed area cannot be well restored due to the information loss of the single LDR image. To address this issue, we introduce color events into the imaging pipeline, which record asynchronous pixel-wise color changes in a high dynamic range, enabling edge-like scene perception under challenging lighting conditions. Specifically, we propose a joint framework that incorporates color events and a single LDR image to restore both content and color of an HDR image, where an exposureaware transformer (EaT) module is designed to propagate the informative hints, provided by the normal-exposed LDR regions and the event streams, to the missing areas. In this module, an exposure-aware mask is estimated to suppress distractive information and strengthen the restoration of the over-/under-exposed regions. To our knowledge, we are the first to use color events to enhance single-exposure HDR imaging. We also contribute corresponding datasets, consisting of synthesized datasets and a real-world dataset collected by a DAVIS346-color camera. The datasets can be found at https://www.kaggle.com/datasets/mengyaocui/ce-hdr. Extensive experiments demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{cui2024color,
  author = {Mengyao Cui and Zhigang Wang and Dong Wang and Bin Zhao and Xuelong Li},
  title = {{Color Event Enhanced Single-Exposure HDR Imaging}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i2.27904},
}

Loading related content...

Figure