Hybrid Lighting Representations for Scene Relighting

16 concepts9 papers1 proposal

When inserting a virtual object into a real-world photograph or relighting a reconstructed 3D scene, we must accurately model how light arrives at every surface point. However, real-world illumination presents a fundamental representation challenge. It is simultaneously low-frequency (smooth ambient light from walls or a cloudy sky) and high-frequency (intense, localized light from a lightbulb or the sun). It also originates from different distances, spanning infinitely distant environments down to local, surface-to-surface inter-reflections.

Historically, researchers relied on single, uniform lighting representations like 2D high-dynamic-range (HDR) environment maps or Spherical Harmonics. However, these uniform approaches force an impractical trade-off: mathematical models that are efficient enough for real-time rendering tend to blur out sharp highlights, while high-resolution pixel maps are computationally heavy, difficult to edit, and fail to capture local, position-dependent lighting changes.

To resolve this tension, a powerful paradigm has emerged: hybrid lighting representations. Instead of forcing a single model to represent all illumination, these techniques decompose lighting into distinct components based on frequency, distance, and mathematical structure.


Decomposing by Frequency: Spherical Harmonics and Spherical Gaussians

The most direct way to split lighting is by its frequency components. Different mathematical representations excel at modeling different parts of the light spectrum:

  • Spherical Harmonics (SH) act like a low-pass filter on a sphere. They require very few coefficients to represent smooth, diffuse background lighting, but require an intractable number of parameters to represent sharp, localized light sources.
  • Spherical Gaussians (SG) act like localized directional bumps on a sphere. They are highly efficient at representing sharp, high-frequency directional light sources using just a few parameters (direction, amplitude, and sharpness).

By combining these two representations, models can capture the full spectrum of real-world illumination without massive computational overhead.

Real-World Illumination
 ├── Low-Frequency Ambient  ---> Spherical Harmonics (SH)  ---> Soft Shadows & Diffuse Glow
 └── High-Frequency Direct  ---> Spherical Gaussians (SG)  ---> Sharp Specular Highlights

For instance, Xu et al. developed a framework that predicts an HDR environment map from a single image by simultaneously estimating SGs for sharp highlights and SHs for ambient light. Similarly, X. Zhu et al. demonstrate that this dual SH-SG parametric model is crucial for realistic object insertion and shadow rendering within Neural Radiance Fields (NeRFs). By assigning each frequency component to its optimal mathematical representation, these frameworks achieve both realistic soft shadows and crisp specular reflections.


Decomposing by Distance: Distant Environment vs. Near-Field Radiance

Lighting does not just vary by frequency; it also varies by where it originates. Distant illumination (like the sky or far-away windows) hits a scene uniformly, whereas near-field illumination (like light bouncing off a table onto an object resting on it) changes dramatically depending on the object's exact position.

To handle highly reflective objects, Liu et al. utilize separate neural networks to decouple distant direct light from local indirect light, preventing complex surface reflections from ruining the geometric reconstruction.

Taking this further, Wu et al. combine smooth mathematical representations (Spherical Gaussians) for distant environment light with a neural radiance cache to capture near-field indirect light transport. This hybrid approach allows for the efficient sampling of global light while preserving the high-frequency details of local surface-to-surface reflections.

This spatial decomposition is equally vital when integrating virtual objects into neural scenes. As Ye et al. show, NeRFs inherently struggle to capture all external illumination sources. By combining estimated distant environment lighting with near-field lighting derived directly from the NeRF volume, the authors ensure that inserted objects receive accurate illumination from both the wider environment and their immediate physical surroundings.


Bridging Parametric Control and Non-Parametric Realism

A final dimension of hybrid lighting involves balancing parametric models (explicit mathematical formulations or 3D shapes) with non-parametric or generative models (textures, image features, or neural networks).

Parametric models provide physical interpretability and energy conservation, which are essential for generating accurate shadows and allowing users to edit the scene. Non-parametric models excel at capturing the rich, complex, and unconstrained textures of the real world.

Several distinct strategies leverage this synergy:

  • Discrete 3D Lights + Ambient Terms: Gardner et al. move away from 2D maps by estimating discrete 3D parametric light sources in physical space alongside a global ambient term. Because the lights have 3D positions, shadows and highlights shift correctly as an object moves through a room.
  • Parametric Shapes + Textures: Weber et al. combine a parametric HDR light source with a non-parametric, low-dynamic-range (LDR) textured cuboid layout. The parametric light handles the heavy lifting of shadow generation, while the textured cuboid provides the complex environmental details needed for realistic reflections.
  • Sky Gradients + Image Features: For outdoor scenes, Yongjie Zhu et al. pair parametric models representing low-frequency sky gradients with non-parametric image features, balancing global atmospheric consistency with sharp local details.
  • Parametric Models + Generative Networks: Dastjerdi et al. integrate parametric Spherical Gaussians with a Generative Adversarial Network (GAN). The parametric component ensures energy-accurate shading for shadows, while the generative network synthesizes high-fidelity textures for sharp reflections. This allows users to intuitively move or edit a light source while the system automatically generates realistic, updated reflections.

By breaking down the monolithic problem of light estimation into specialized, cooperating representations, hybrid lighting models successfully bridge the gap between computational efficiency, user editability, and photorealistic rendering.

Key Foundational Work

Spatially-Varying Outdoor Lighting Estimation From Intrinsics (2021) — contributed 2 concept elements to this direction.
Deep Parametric Indoor Lighting Estimation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_lighting_representation_924ea7aa,
  title = {hybrid lighting representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-lighting-representation/rdr_924ea7aa3d0140a6917fe957e873af5c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_lighting_representation_924ea7aa_v1,
  title = {hybrid lighting representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-lighting-representation/rdr_924ea7aa3d0140a6917fe957e873af5c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake