Abstract
This paper addresses the task of estimating the light arriving from all directions to a 3D point observed at a selected pixel in an RGB image. This task is challenging because it requires predicting a mapping from a partial RGB observation by a camera to a complete illumination map for a different 3D point, which depends on the 3D location of the selected pixel, the distribution of unobserved light sources, the occlusions by scene geometry, etc. Previous methods attempt to learn this complex mapping directly using a single black-box neural network which often fails to estimate high-frequency lighting details for scenes with complicated 3D geometry. Instead, we propose "Neural Illumination," a new approach that decomposes illumination prediction into several simpler differentiable sub-tasks: 1) geometry estimation, 2) scene completion, and 3) LDR-to-HDR estimation. The advantage of this approach is that the sub-tasks are relatively easy to learn and can be trained with direct supervision, while the whole pipeline is fully differentiable and can be fine-tuned with end-to-end supervision. Experiments show that our approach performs significantly better quantitatively and qualitatively than prior work.
Introduction
Realistic object insertion in Mixed Reality requires an accurate understanding of the lighting environment at specific 3D points. This "illumination map" describes the light arriving from all directions to a target locale. However, predicting this from a single low-dynamic range (LDR) image is a formidable challenge. The camera typically captures only a partial view of the scene, missing light sources behind it or occluded by geometry. Furthermore, lighting is spatially varying; the illumination at a table's surface differs significantly from that in a shadowed corner.
Previous attempts to solve this have largely relied on "black-box" neural networks. While these models can estimate general lighting, they often struggle with high-frequency details, resulting in blurry reflections and inaccurate highlights. This paper introduces "Neural Illumination," a framework that overcomes these limitations by decomposing the task into interpretable, differentiable sub-modules.
The Neural Illumination Framework
The core philosophy of Neural Illumination is that predicting light is significantly easier when the underlying scene geometry is understood. The proposed pipeline consists of four primary stages:
- Geometry Estimation: Using a U-Net architecture, the model first predicts the 3D structure of the visible scene, specifically surface normals and plane offsets. This provides the spatial context needed to relate pixels in the input image to the 3D world.
- Geometry-aware Warping: With the predicted geometry, the system "warps" visible pixels from the input image onto a spherical projection centered at the target 3D locale. This step ensures that observed light sources are placed in their correct geometric positions in the final illumination map.
- LDR Panorama Completion: Since the camera only sees part of the room, the warped projection contains "holes." A Generative Adversarial Network (GAN) based on a ResNet-50 architecture is employed to fill these gaps, hallucinating plausible lighting for unobserved regions.
- LDR-to-HDR Estimation: Finally, the system maps LDR colors to high-dynamic range (HDR) intensities. This module uses a specialized diffuse convolution loss to ensure that the predicted high-intensity light sources produce realistic reflections when applied to virtual materials.
Dataset and Training
To train such a complex modular system, the authors developed the Neural Illumination Dataset. Building upon the Matterport3D dataset—which contains nearly 200,000 HDR RGB-D panoramas—the researchers used depth data to re-render these scenes from tens of thousands of different 3D perspectives. This effort produced over 90,000 ground truth illumination maps paired with 129,600 LDR input images, providing a robust foundation for learning spatially varying lighting.
Results and Evaluation
Neural Illumination demonstrates significant improvements over prior state-of-the-art methods. Quantitatively, the model achieved an $\ell_2$ (log) error of 0.202, significantly outperforming previous benchmarks like Gardner et al., which scored 0.375.
Qualitatively, the benefits are even more apparent. When rendering virtual objects like shiny teapots or spheres, the Neural Illumination maps produce sharp, high-frequency reflections of windows and lamps. In contrast, black-box models typically yield dull, muddy reflections that fail to capture the "sparkle" of real-world lighting. Ablation studies confirmed that the inclusion of GAN-based completion and the diffuse convolution loss were essential for achieving these high-fidelity results.
Conclusion
By breaking down the complex problem of illumination prediction into geometry estimation, warping, and completion, Neural Illumination provides a more accurate and physically grounded approach to lighting estimation. While the model still faces challenges in perfectly hallucinating completely hidden light sources, it represents a major step forward for Mixed Reality applications, enabling more realistic and spatially consistent virtual object insertion. Future work may involve explicit modeling of surface materials to further refine how light interacts with the environment.
Summary updated