Abstract
High dynamic range (HDR) illumination estimation from a single low dynamic range (LDR) image is a significant task in computer vision, graphics, and augmented reality. We present a two-stage deep learning-based method to predict an HDR environment map from a single narrow field-of-view LDR image. We first learn a hybrid parametric representation that sufficiently covers high- and low-frequency illumination components in the environment. Taking the estimated illuminations as guidance, we build a generative adversarial network to synthesize an HDR environment map that enables realistic rendering effects. We specifically consider the rendering effect by supervising the networks using rendering losses in both stages, on the predicted environment map as well as the hybrid illumination representation. Quantitative and qualitative experiments demonstrate that our approach achieves lower relighting errors for virtual object insertion and is preferred by users compared to state-of-the-art methods.
Introduction
In Augmented Reality (AR) and computer graphics, "relighting" is the process of making a virtual object look like it truly belongs in a real-world photo. To achieve visual consistency, an HDR environment map—a 360° panorama of all incident light—is required. However, most users capture only a single, standard LDR photo with a limited field of view. Recovering the full 360° HDR environment from such limited data is an under-constrained problem, as the model must infer light sources outside the camera's view and intensities beyond the sensor's range. Existing methods often struggle with blurry results or fail to capture sharp highlights and subtle ambient lighting.
Methodology
The proposed framework employs a novel two-stage architecture designed to capture both high-frequency and low-frequency lighting components.
1. Hybrid Illumination Regression
Instead of directly synthesizing an image, the model first estimates a hybrid parametric representation. This approach recognizes that parametric models like Spherical Harmonics (SH) are excellent for soft shadows but poor for highlights, while Spherical Gaussians (SG) excel at the opposite. By combining them, the model covers the full frequency spectrum of light.
- Spherical Gaussians (SG): These represent high-frequency light sources (e.g., lamps, windows). A Transformer-based network is used to predict these parameters, leveraging the Transformer's ability to globally reason about spatial relationships between sparse light sources based on visible scene context.
- Spherical Harmonics (SH): These represent low-frequency, ambient light, typically handled by a standard CNN backbone.
These parameters are projected into guidance maps—a Gaussian lighting map and a diffuse irradiance map—which inform the subsequent synthesis stage.
2. Generative Environment Map Synthesis
A Generative Adversarial Network (GAN) takes the original LDR image and the guidance maps to synthesize the final 360° HDR environment map. Spherical convolutions are utilized to account for the geometric distortions inherent in equirectangular panoramic images.
3. Rendering-Aware Loss
A key innovation is the "rendering-aware" loss function. Standard pixel-wise losses (like MSE) often fail to capture the visual impact of lighting errors; a small pixel error in a light source can cause a significant visual artifact in a rendered object. The authors introduced a differentiable rendering layer that allows the model to "render" a virtual object during training. By comparing the rendered result of the predicted map against the ground truth, the network is forced to prioritize lighting features that are most critical for visual realism.
Experiments and Results
The model was trained using the Laval Indoor HDR Dataset, with nearly 20,000 training pairs generated to simulate real-world user photos. The SVBRDF Dataset provided material textures (albedo, normals, roughness) for the rendering layer.
Quantitative Performance
The method significantly outperformed state-of-the-art models (e.g., Gardner et al., Zhan et al.) across key metrics:
- Relighting RMSE: 0.0494 (vs. 0.0741 for the next best).
- Angular Error: 7.35 (vs. 10.31 for the next best).
Qualitative Evaluation
In a user study, participants preferred this method's results 39.6% of the time when compared against ground truth and four other competitors—the highest preference rate among all tested methods. Visually, the model produces sharp specular highlights and realistic soft shadows, even when primary light sources are located behind the camera.
Conclusion
By combining a hybrid lighting representation with a rendering-aware loss, this framework effectively bridges the gap between single-image LDR capture and realistic HDR relighting. While the current model excels in indoor scenes, future work may involve incorporating scene semantics and more complex geometry to further enhance the fidelity of the predicted environment maps in diverse environments.
Summary updated