Deep HDR Hallucination for Inverse Tone Mapping

Demetris Marnerides, Thomas Bashford-Rogers, Kurt Debattista

TL;DR

This paper introduces a GAN-based approach to Inverse Tone Mapping (ITM) that doesn't just stretch the dynamic range of a photo—it actually "hallucinates" missing details in over- and under-exposed regions. By combining a novel HDR-specific pixel normalization technique with a clever data augmentation strategy, the authors achieve state-of-the-art results in reconstructing high-quality HDR images from single LDR inputs.

What Problem Does This Paper Address?

Most of the photos we take and view are Low Dynamic Range (LDR). They have a limited range of brightness, which often leads to "clipping": bright skies become pure white (over-exposed) and deep shadows become pure black (under-exposed). Inverse Tone Mapping (ITM) is the process of recovering High Dynamic Range (HDR) information from these LDR images.

The challenge is that once a pixel is clipped to pure white or black, the original information is gone. Existing deep learning methods are relatively good at expanding the range of "well-exposed" areas, but they struggle to fill in the blanks in clipped regions, often producing flat, blurry, or unrealistic results. This paper tackles the "hallucination" problem—reconstructing plausible textures and details where the data has been completely lost.

Methods

The authors propose a system called Deep HDR Hallucination (DHH). Its architecture and training regime feature several key innovations:

  • GAN-Based Architecture: Unlike previous models that use standard regression losses (which tend to produce "average," blurry results), DHH uses a Generative Adversarial Network. A UNet-style generator creates the HDR image, while a PatchGAN discriminator tries to distinguish it from real HDR data. This encourages the model to produce sharp, realistic textures.
  • Hybrid Loss Function: The model is optimized using a combination of Hinge Loss (for GAN stability), Perceptual Loss (using VGG features to ensure visual similarity), and Feature Matching Loss (to ensure structural consistency).
  • HDR Pixel Distribution Transform: This is a standout contribution. Linear HDR values are exponentially distributed, with most pixels clustered near zero. This "skew" makes it very hard for neural networks to learn. The authors proposed a mathematical transform to map these values into a more manageable Gaussian (normal) distribution before feeding them into the network.
  • Instance Normalization: Through ablation studies, the authors found that Instance Normalization preserves contrast and hallucination quality better than the more common Batch Normalization, which tended to "average out" the results into desaturated tones.

Datasets

One of the biggest hurdles in HDR research is the lack of massive datasets. While LDR datasets have millions of images, high-quality HDR datasets often have fewer than 1,000.

To solve this, the authors developed an HDR Data Augmentation method. They took a large LDR dataset from Flickr (45,166 images) and used a pre-trained "teacher" model (GUNet) to generate synthetic HDR targets. They then intentionally "culled" (clipped) the top and bottom 10% of values to create the LDR inputs. This allowed them to train the "hallucination" network on a massive variety of scenes, from landscapes to cityscapes.

Key Insights and Ideas

  • Hallucination is Ill-Posed: There isn't one "correct" way to fill in a clipped white sky. It could have been clear, or it could have had wispy clouds. GANs are uniquely suited for this because they can learn the distribution of possible realities rather than just trying to find a single mathematical average.
  • Normalization is the Secret Sauce: The "Aha!" moment here is the HDR Pixel Transform. By recognizing that the raw data distribution was fighting the network's internal math, the authors simplified the optimization problem significantly.
  • Skip Connections are Vital: The paper notes that architectures without skip connections (like simple autoencoders) fail to reproduce well-exposed areas faithfully, as they lose the fine-grained spatial information required for high-fidelity reconstruction.

Key Results

  • Qualitative Superiority: The DHH model successfully reconstructs textures in clouds and ground cover that other state-of-the-art models (like ExpandNet or GUNet) leave as flat, uniform blobs.
  • Quantitative Competitiveness: Even though standard metrics like PSNR and SSIM don't "reward" hallucinated content (since the hallucination might not match the ground truth pixel-for-pixel), DHH remained quantitatively competitive with top-tier models across metrics like HDR-VDP-2 and PU-PSNR.
  • Resolution Consistency: The model maintains consistent dynamic range expansion across different input resolutions (256px to 1024px), though the specific structure of the hallucinations can shift slightly as the input size changes.

Limitations and Future Work

While the model produces plausible textures, it does not always reconstruct semantic details perfectly. For example, it might fill a bright spot with realistic cloud textures but fail to reconstruct the specific circular shape of the sun.

The authors suggest that future work could involve:

  1. Temporal Consistency: Extending the model to HDR video to ensure hallucinations don't "flicker" between frames.
  2. Multi-Task Learning: Combining ITM with other tasks like colorization, de-noising, or super-resolution.
  3. Semantic Guidance: Incorporating object detection or segmentation to help the model "know" what it is hallucinating (e.g., "this is a sky, use cloud textures; this is a road, use asphalt textures").

Summary updated

Abstract

Inverse Tone Mapping (ITM) methods attempt to reconstruct High Dynamic Range (HDR) information from Low Dynamic Range (LDR) image content. The dynamic range of well-exposed areas must be expanded and any missing information due to over/under-exposure must be recovered (hallucinated). The majority of methods focus on the former and are relatively successful, while most attempts on the latter are not of sufficient quality, even ones based on Convolutional Neural Networks (CNNs). A major factor for the reduced inpainting quality in some works is the choice of loss function. Work based on Generative Adversarial Networks (GANs) shows promising results for image synthesis and LDR inpainting, suggesting that GAN losses can improve inverse tone mapping results. This work presents a GAN-based method that hallucinates missing information from badly exposed areas in LDR images and compares its efficacy with alternative variations. The proposed method is quantitatively competitive with state-of-the-art inverse tone mapping methods, providing good dynamic range expansion for well-exposed areas and plausible hallucinations for saturated and under-exposed areas. A density-based normalisation method, targeted for HDR content, is also proposed, as well as an HDR data augmentation method targeted for HDR hallucination.
BibTeX
@inproceedings{marnerides2021deep,
  author = {Demetris Marnerides and Thomas Bashford-Rogers and Kurt Debattista},
  title = {{Deep HDR Hallucination for Inverse Tone Mapping}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=Xk_I9EIVmc},
}

Loading related content...

Figure