When rendering a virtual object into a real-world photograph, the standard approach has long relied on Image-Based Lighting (IBL) via 2D environment maps. By capturing a 360-degree panorama of a room, you can project that texture onto an infinite sphere surrounding your virtual object, treating each pixel as a distant light source.
This assumption works beautifully for outdoor landscapes where the sky is effectively at infinity. However, it fails completely in indoor or near-field environments. If you move a virtual object across a room, its relationship to local light sources—like a desk lamp or a nearby window—should change. The shadows should rotate, highlights should shift, and the light intensity should fall off as the object moves away from the source.
As Yiyu Zhuang et al. observe in A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields, traditional 2D lighting representations lack spatial awareness; they cannot model how physical objects within a 3D environment block or filter light before it reaches a surface. Furthermore, Pratul P. Srinivasan et al. point out in Lighthouse: Predicting Lighting Volumes for Spatially-Coherent Illumination that estimating independent 2D light maps for different points in a scene leads to severe spatial inconsistencies and immersion-breaking flickering.
To achieve realistic, spatially-varying illumination, researchers have shifted from 2D directional maps to 3D spatially-aware representations. By embedding depth and volumetric structure directly into the illumination model, rendering engines can compute the exact light arriving at any specific 3D coordinate.
From Pixels to Primitives: Parametric 3D Light Sources
One intuitive way to introduce spatial awareness is to explicitly extract light sources as physical 3D objects. Instead of treating a lamp as a bright patch of pixels on a distant sphere, we can treat it as a geometric entity with a defined 3D position, size, and intensity fall-off.
Marc-André Gardner et al. pioneered this shift in Deep Parametric Indoor Lighting Estimation by training a deep neural network to estimate a set of discrete 3D "parametric" lights from a single image. Because these light sources are defined by their physical distance and angular size, light intensity and shadow directions automatically adapt as a virtual object moves relative to them.
Traditional 2D Environment Map (Infinite Distance)
[ Light Source Pixel ] --------(Infinite Ray)--------> [ Virtual Object ]
Result: Moving the object does not change shadow direction or intensity.
3D Parametric / Volumetric Lighting (Spatially Aware)
[ 3D Light Source ] ---(Distance Fall-off & Occlusion)---> [ Virtual Object ]
Result: Moving the object correctly shifts shadows and modulates brightness.
Building on this concept, Zhengqin Li et al. demonstrated in Physically-Based Editing of Indoor Scene Lighting from a Single Image that representing light sources as parametric 3D primitives enables direct scene manipulation. Users can physically remove or adjust individual lights, and the system will automatically compute consistent updates for shadows and global interreflections.
To better map complex, multi-source environments, Fangneng Zhan introduces GMLight: Lighting Estimation via Geometric Distribution Approximation, which models lighting as a discrete distribution over 3D anchor points. By incorporating scene depth, this approach accounts for the physical distance between light sources and objects. Similarly, Jiayang Bai et al. developed Deep Graph Learning for Spatially-Varying Indoor Lighting Prediction, leveraging graph convolutional networks to inject depth values into spherical light representations, ensuring that shading and shadows change dynamically based on the object's 3D position.
Grounding the Map: Depth-Augmented Radiance Caches
If we want to preserve the dense, high-frequency detail of panoramic images while overcoming their spatial limitations, we must anchor the lighting data to real-world geometry.
An early solution proposed by Dalai Felinto et al. in Production Framework for Full Panoramic Scenes with Photorealistic Augmented Reality introduces the light-depth environment map. By pairing every high-dynamic-range (HDR) pixel with a corresponding spatial depth value, the rendering engine can treat pixels as localized point light sources rather than distant directional vectors. This allows for accurate local shadows and reflections that naturally vary with object proximity.
Alternatively, this radiance data can be cached directly onto the 3D geometry of the scene itself. Manuele Sabbadin et al. demonstrated in High Dynamic Range Point Clouds for Real-Time Relighting that treating 3D point clouds as spatially-aware radiance and visibility caches allows rendering engines to compute complex local illumination and occlusion effects, bypassing the limitations of traditional environment maps entirely.
Volumetric Lighting Fields: Continuous 3D Illumination
Instead of relying on discrete light primitives or surface-bound caches, the most robust way to ensure spatial coherence across an entire environment is to model lighting as a continuous 3D volume.
In their framework Lighthouse, Pratul P. Srinivasan et al. proposed predicting a shared, multiscale 3D volumetric representation of the environment's lighting from a narrow-baseline stereo pair. Because all local lighting information is derived from this single, unified volume, virtual objects can be placed at any coordinate in space while maintaining perfect spatial coherence and avoiding visual flickering.
[ 3D Voxel / Radiance Volume ]
+-------------------------+
| (x1,y1,z1) -> Light A | ----> Inserts Object 1 (Correct Local Light)
| (x2,y2,z2) -> Light B | ----> Inserts Object 2 (Correct Local Light)
+-------------------------+
This volumetric paradigm integrates naturally with modern inverse rendering and neural fields:
- Voxel Volumes: Junyong Choi et al. developed MAIR: Multi-View Attention Inverse Rendering with 3D Spatially-Varying Lighting Estimation, which represents scene lighting as a 3D volume of voxels. This accounts for both direct and indirect light transport, enabling consistent object insertion at any height or position within the reconstructed space.
- Neural Radiance Fields: Yiyu Zhuang et al. model the environment as a volumetric radiance field in A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields. By tracing rays through a 3D density and color distribution, they naturally capture near-field lighting effects and local occlusions. Similarly, Zhan Lu et al. developed Pano-NeRF, which integrates the rendering equation directly into neural radiance fields to estimate spatially-varying lighting throughout a 3D volume, recovering blown-out highlights without requiring pre-trained HDR models.
Per-Primitive Neural Lighting
With the advent of explicit 3D representations like Gaussian Splatting, researchers have begun embedding learnable lighting properties directly into individual 3D primitives. This allows the geometry itself to dictate how light is received, modulated, and scattered.
In VoLux-GAN, Feitong Tan et al. showed that integrating diffuse and specular lighting components directly within a volumetric rendering framework—accumulating contributions along 3D rays using local surface normals—enables seamless relighting under arbitrary HDR maps.
For real-time view synthesis under challenging conditions, Sheng Ye et al. introduced Gaussian in the Dark: Real-Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting. They assign learnable light features to individual 3D primitives, which are rasterized and refined by a neural network to produce a modulation map. This map adjusts the base radiance to capture complex local highlights and shadows that global exposure models miss.
To handle complex light-material interactions, Iris Zheng et al. developed SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting. By tracing rays directly through 3D volumetric primitives rather than relying on 2D screen-space approximations, their method produces physically consistent soft shadows that correctly respect the 3D geometry during dynamic relighting.
Finally, L.M. Cheng et al. demonstrated in 3D Gaussian Splatting Model for Low-Light Enhancement that a lightweight neural network can predict local illumination factors for 3D primitives based on their spatial coordinates and the observer's viewing direction. This decouples intrinsic scene color from lighting, enabling complex, view-dependent lighting effects to be adjusted in real time.
Key Foundational Work
Deep Parametric Indoor Lighting Estimation (2019) — contributed 2 concept elements to this direction.
Production Framework for Full Panoramic Scenes with Photorealistic Augmented Reality (2013) — contributed 2 concept elements to this direction.
High Dynamic Range Point Clouds for Real-Time Relighting (2019) — contributed 1 concept elements to this direction.