Deep Joint Demosaicing and High Dynamic Range Imaging within a Single Shot

Yilun Xu, Ziyang Liu, Xingming Wu, Weihai Chen, Changyun Wen, Zhengguo Li

Abstract

Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over- and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-to-end single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods.

Introduction

High Dynamic Range (HDR) imaging is essential for capturing scenes with high contrast. Traditional methods rely on "exposure stacking," which merges multiple frames taken at different exposures. However, this approach is prone to "ghosting" artifacts when subjects or the camera move between captures. Single-shot HDR imaging using Spatially Varying Exposure (SVE) offers a solution by capturing different exposure levels within a single Bayer image. While this eliminates ghosting, it introduces significant reconstruction challenges due to sparse data and ill-exposed pixels.

The SVE Reconstruction Challenge

In an SVE sensor, pixels are interleaved with different exposure times (e.g., short exposure $\tau_s$ and long exposure $\tau_l$). This creates a complex pattern where only a fraction of the pixels for any specific exposure and color are available.

The SVE Bayer pattern captures different exposure levels in a single frame, but results in sparse data for reconstruction.

Furthermore, many pixels in these raw captures are "ill-exposed"—either saturated in long exposures or buried in noise in short exposures. Traditional sequential pipelines that perform demosaicing before HDR reconstruction often suffer from cumulative errors, where artifacts from the first stage are amplified in the second.

Proposed Methodology: Joint Demosaicing and HDRI

To address these issues, we propose a one-stage deep learning framework that performs demosaicing and HDR reconstruction simultaneously. The architecture is built around two key innovations:

  1. Spatially Varying Convolution (SVC): Unlike standard convolutions that use uniform kernels, SVC employs eight distinct sets of kernels that cycle through the image to match the 8-pixel periodicity of the SVE Bayer pattern (4 colors $\times$ 2 exposures). This ensures the network correctly interprets the specific color and exposure of each pixel.
  2. Exposure-Guidance Branch (EGB): To handle unreliable data, an exposure-guidance mask is generated to identify ill-exposed pixels. The EGB processes this mask to provide "prior knowledge" to the main Reconstruction Branch (RB), allowing the model to intelligently compensate for missing or noisy information rather than simply filtering it out.
The proposed framework integrates the Reconstruction Branch (RB) with an Exposure Guidance Branch (EGB) to produce full-resolution HDR images from raw SVE Bayer data.

Experimental Results

The framework was evaluated using two datasets: VETHDR-Nikon and a newly introduced VETHDR-Canon dataset, which provides high-quality 16-bit ground truths.

Quantitative Performance

The proposed model outperformed state-of-the-art methods across standard metrics. On the Nikon dataset, it achieved an HDR-VDP score of 66.02, surpassing the previous best of 65.81. It also demonstrated superior performance in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), indicating higher fidelity in both luminance and structure.

Qualitative Analysis

Visual comparisons reveal that the joint approach effectively eliminates the horizontal striping and color distortions common in sequential methods. The model recovers sharp textures and fine details in both highlight and shadow regions, where other algorithms often produce blur or artifacts.

Conclusion

By merging demosaicing and HDR imaging into a single end-to-end process, this framework avoids the pitfalls of cumulative errors in traditional pipelines. The combination of Spatially Varying Convolutions and Exposure Guidance allows for high-quality, full-resolution HDR reconstruction from a single shot, effectively solving the ghosting problem. Future work will focus on extending this framework to HDR video and optimizing it for real-time performance on mobile hardware.

Summary updated

Abstract

Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying exposures. For the latter one, an exposure-guidance method is proposed against the interference from over- and under-exposed pixels. Finally, a joint demosaicing and HDRI deep learning framework is formalized to include the two novel components and to realize an end-to-end single-shot HDRI. Experiments indicate that the proposed end-to-end framework avoids the problem of cumulative errors and surpasses the related state-of-the-art methods.
BibTeX
@article{xu2022deep,
  author = {Yilun Xu and Ziyang Liu and Xingming Wu and Weihai Chen and Changyun Wen and Zhengguo Li},
  title = {{Deep Joint Demosaicing and High Dynamic Range Imaging within a Single Shot}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2022},
  url = {https://openreview.net/forum?id=SXe26DI_G0},
}
Figure