Abstract
Standard digital cameras often fail to capture the full range of light in a scene, resulting in "blown-out" highlights where details are lost to pure white pixels. This paper presents a novel deep learning approach to reconstruct High Dynamic Range (HDR) images from a single Low Dynamic Range (LDR) input. By introducing a feature masking mechanism to ignore invalid saturated data and a perceptual loss function to synthesize realistic textures, the proposed model "hallucinates" missing details with high fidelity. Pre-training on image inpainting and a targeted patch-sampling strategy allow the network to outperform previous methods, generating artifact-free, visually pleasing highlights.
Introduction
Single-image HDR reconstruction is an inverse problem aimed at recovering information lost during the camera's capture process. Standard convolutional neural networks (CNNs) face two primary hurdles in this task. First, they are "validity-blind," applying the same filters to both well-exposed and saturated (invalid) pixels, which often results in checkerboard artifacts and halos at the boundaries of bright regions. Second, reliance on pixel-wise loss functions like Mean Squared Error (MSE) tends to produce blurry, textureless results. This work addresses these challenges by treating highlight reconstruction as a specialized form of image inpainting, leveraging masked convolutions and perceptual losses to restore complex textures.
Methodology
The core of the proposed system is a U-Net architecture enhanced by several innovations designed to handle the ambiguity of saturated regions.
Feature Masking
To prevent saturated pixels from corrupting the features of valid pixels, the authors implement a "soft mask" ($M$). At each layer, feature maps are multiplied by this mask. Crucially, the mask is updated at every layer based on the filter weights, allowing the network to automatically track which features are "valid" as they propagate deeper into the network.
Perceptual and Style Loss
Instead of simple pixel comparisons, the model utilizes VGG-based perceptual losses adapted for the HDR domain. A $\mu$-law range compressor maps HDR values into a standard range compatible with the VGG network, forcing the model to produce results that are perceptually similar to real-world textures rather than just numerical averages.
Training Strategy
Since high-quality HDR training data is scarce, the model undergoes a two-stage training process. It is first pre-trained on a massive image inpainting task using the MIT Places dataset (~2.5 million images). This teaches the network general texture synthesis. Subsequently, it is fine-tuned on a custom HDR dataset. To combat the "boring highlight" bias—where networks learn to produce smooth white blobs—a bilateral filter-based sampling strategy is used to select training patches with high texture complexity in saturated areas.
Datasets and Evaluation
The model was fine-tuned on a custom collection of 735 HDR images and 34 HDR videos, yielding 100,000 high-detail patches. Evaluation was performed on a test set of 75 HDR images, alongside real-world photos from the Google HDR+ dataset and Canon 5D Mark IV captures.
Results and Discussion
The proposed method demonstrates significant improvements over state-of-the-art models such as those by Endo et al. and Eilertsen et al. Quantitatively, the model achieved a Mean Square Error of 0.0356 and a superior HDR-VDP-2 score of 63.18.
Qualitatively, the method successfully reconstructs sharp details in window reflections and cloud textures, effectively eliminating the "halo" effect common in previous approaches. The use of feature masking ensures the network relies on valid contextual information to fill in missing data, while the inpainting pre-training provides a robust foundation for synthesizing realistic details.
Limitations and Future Work
While effective, the model can struggle with extremely intricate, unique textures where context is insufficient for accurate hallucination. Color bleeding from adjacent objects into neutral highlights remains a minor issue. Future work will focus on temporal stability for video reconstruction, potentially through recurrent components or temporal regularization to prevent frame-by-frame flickering.
Summary updated