In the quest to create photorealistic 3D digital twins, researchers have moved beyond merely capturing how a scene looks to understanding why it looks that way. Traditional neural representations like Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) often "bake" lighting, geometry, and material properties into a single, inseparable representation. This makes it impossible to change the time of day or insert a virtual object without it looking like a flat, poorly composited sticker.
The emerging solution across recent literature is a hybrid decomposition strategy. Instead of using one mathematical tool to describe all light, researchers are splitting appearance into distinct frequency bands and spatial zones, using different representations for each.
The Frequency Split: Smooth Skies and Sharp Highlights
The most common pattern in this research direction is the decomposition of lighting into low-frequency and high-frequency components.
Low-frequency lighting—the soft, ambient glow of a cloudy sky or the general "mood" of a room—is efficiently captured by Spherical Harmonics (SH). SH functions act like a low-pass filter; they are computationally cheap and excellent at representing smooth gradients. However, they fail miserably at capturing sharp details like the glint of a sun reflection or the crisp edge of a shadow.
To solve this, researchers like Fangneng Zhan in MixLight and Kai Xu 0004 in Relighting Scenes with Object Insertions augment SH with Spherical Gaussians (SG). While SH handles the "blurry" ambient light, SGs act as directional "lobes" that can represent intense, localized light sources. This hybrid approach allows a model to maintain the overall atmospheric tone while still producing sharp specular highlights.
Surface Detail vs. Global Illumination
A similar frequency split is applied to the surfaces themselves. In vanilla 3D Gaussian Splatting, each "splat" has a single color. To capture a high-resolution texture, you need millions of tiny Gaussians, which is memory-intensive.
B. T. Chao and colleagues introduce Textured Gaussians, which decouple these properties. They use SH for the view-dependent, low-frequency lighting and "tiny, local texture maps" for the high-frequency surface details. By allowing each Gaussian to carry its own micro-texture, the model can represent intricate patterns (like the weave of a fabric or the grain of wood) without needing an astronomical number of primitives. This concept is further refined by Sheng-chi Hsu in A$^2$TG, which uses adaptive, non-square textures to optimize memory even further.
Bridging Near-Field and Far-Field
Another critical facet of this direction is the spatial decomposition of light. For an object to look "real" in a scene, it must be lit by both the distant environment (the sun, the sky) and the nearby geometry (reflections from a wall, shadows from a nearby chair).
- Far-field lighting is typically modeled as an environment map or a parametric sky model. Zian Wang and the team at NVIDIA demonstrate in Neural Light Field Estimation that separating a high-dynamic range (HDR) sky model from local volumetric representations is essential for capturing extreme solar intensity alongside subtle local shadows.
- Near-field lighting requires understanding the 3D structure around the object. Chongyang Ma in LitAR handles this by using dense point clouds for nearby elements and directional data for distant ones.
By combining these, as seen in the work of Keyang Ye on Real-time Inserting Virtual Objects, virtual objects can receive consistent illumination from both visible and out-of-view sources, preventing them from appearing unnaturally dark or disconnected from their surroundings.
Why This Works: The Best of Both Worlds
The intuition behind this research direction is a "divide and conquer" philosophy.
- Parametric models (like SH and SG) provide stability, physical correctness (energy conservation), and easy editability. They are the "skeleton" of the lighting.
- Non-parametric models (like textures, GAN-generated features, or neural shadow factors) provide the "flesh"—the complex, messy details of the real world that math formulas can't easily describe.
As shown in EverLight by Mohammad Reza Karimi Dastjerdi, this synergy allows for precise manipulation—like moving a sun lamp—while the generative components automatically fill in the realistic reflections and shadows that should follow. By assigning specific frequency bands and spatial zones to the representations best suited for them, the field is moving toward 3D scenes that are not just pictures, but interactive, relightable worlds.