Deep HDR Hallucination for Inverse Tone Mapping

Demetris Marnerides, Thomas Bashford-Rogers, Kurt Debattista

Abstract

Inverse Tone Mapping (ITM) methods attempt to reconstruct High Dynamic Range (HDR) information from Low Dynamic Range (LDR) image content. The dynamic range of well-exposed areas must be expanded and any missing information due to over/under-exposure must be recovered (hallucinated). The majority of methods focus on the former and are relatively successful, while most attempts on the latter are not of sufficient quality, even ones based on Convolutional Neural Networks (CNNs). This work presents a GAN-based method that hallucinates missing information from badly exposed areas in LDR images. The proposed method is quantitatively competitive with state-of-the-art inverse tone mapping methods, providing good dynamic range expansion for well-exposed areas and plausible hallucinations for saturated and under-exposed areas. A density-based normalisation method, targeted for HDR content, is also proposed, as well as an HDR data augmentation method targeted for HDR hallucination.

Introduction: The Challenge of Hallucination

Most digital imagery is captured in Low Dynamic Range (LDR), which often results in "clipping"—where bright skies become pure white and deep shadows become pure black. Inverse Tone Mapping (ITM) aims to recover the original High Dynamic Range (HDR) information. While existing deep learning methods excel at expanding the range of well-exposed pixels, they often fail to reconstruct missing textures in clipped regions, leading to flat or blurry artifacts. This paper introduces Deep HDR Hallucination (DHH), a framework designed to "hallucinate" plausible details where data has been lost.

Methodology

The DHH system utilizes a Generative Adversarial Network (GAN) architecture to move beyond standard regression losses, which typically produce "average" and blurry results.

GAN-Based Architecture

The generator follows a U-Net style architecture with skip connections, which are vital for preserving fine-grained spatial information. A PatchGAN discriminator is employed to ensure the generated textures are indistinguishable from real HDR data. The model is optimized using a hybrid loss function comprising Hinge Loss for stability, Perceptual Loss (VGG-based) for visual similarity, and Feature Matching Loss for structural consistency.

HDR Pixel Distribution Transform

A core innovation of this work is the handling of HDR data distributions. Linear HDR values are typically exponentially distributed, with a heavy skew toward zero. This distribution is difficult for neural networks to optimize. The authors proposed a mathematical transform to map these values into a more manageable Gaussian distribution.

HDR pixel density transform: The red histograms show the original skewed distribution, while the blue histograms show the transformed Gaussian-like distribution.

Normalization and Stability

Ablation studies revealed that Instance Normalization outperforms Batch Normalization in this context. While Batch Normalization tended to produce desaturated, averaged results, Instance Normalization better preserved contrast and the quality of hallucinated textures.

Data Augmentation for HDR

Training deep models for HDR is hindered by the scarcity of large-scale HDR datasets. To overcome this, the authors developed a novel data augmentation strategy. By using a pre-trained "teacher" model (GUNet) on a large LDR dataset (Flickr, ~45k images), they generated synthetic HDR targets. They then intentionally clipped the top and bottom 10% of values to create LDR inputs, providing the network with a vast variety of scenes to learn the hallucination process.

Results and Discussion

The DHH model demonstrates a superior ability to reconstruct textures in clouds and ground cover compared to other state-of-the-art models like ExpandNet or GUNet, which often leave these areas as uniform blobs.

Visual results demonstrating the model's ability to hallucinate textures in over-exposed regions across various scenes.

Quantitatively, DHH remains competitive on metrics such as HDR-VDP-2 and PU-PSNR. Although these metrics do not explicitly reward hallucination (as the generated content may not match the ground truth pixel-for-pixel), the model's performance indicates high-fidelity reconstruction. The model also maintains consistency across different input resolutions, though the specific structure of hallucinations may vary slightly.

Conclusion and Future Work

While DHH successfully generates plausible textures, it occasionally struggles with specific semantic details, such as the exact circular shape of a clipped sun. Future research directions include extending the model to ensure temporal consistency in HDR video, integrating multi-task learning (e.g., de-noising or super-resolution), and employing semantic guidance to inform the hallucination process based on scene content.

Summary updated

Abstract

Inverse Tone Mapping (ITM) methods attempt to reconstruct High Dynamic Range (HDR) information from Low Dynamic Range (LDR) image content. The dynamic range of well-exposed areas must be expanded and any missing information due to over/under-exposure must be recovered (hallucinated). The majority of methods focus on the former and are relatively successful, while most attempts on the latter are not of sufficient quality, even ones based on Convolutional Neural Networks (CNNs). A major factor for the reduced inpainting quality in some works is the choice of loss function. Work based on Generative Adversarial Networks (GANs) shows promising results for image synthesis and LDR inpainting, suggesting that GAN losses can improve inverse tone mapping results. This work presents a GAN-based method that hallucinates missing information from badly exposed areas in LDR images and compares its efficacy with alternative variations. The proposed method is quantitatively competitive with state-of-the-art inverse tone mapping methods, providing good dynamic range expansion for well-exposed areas and plausible hallucinations for saturated and under-exposed areas. A density-based normalisation method, targeted for HDR content, is also proposed, as well as an HDR data augmentation method targeted for HDR hallucination.
BibTeX
@inproceedings{marnerides2021deep,
  author = {Demetris Marnerides and Thomas Bashford-Rogers and Kurt Debattista},
  title = {{Deep HDR Hallucination for Inverse Tone Mapping}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=Xk_I9EIVmc},
}
Figure