When reconstructing a 3D scene from images, an algorithm must untangle three interconnected properties: geometry (where surfaces are), materials (how surfaces reflect light), and illumination (where light comes from). Attempting to estimate all three simultaneously creates a fundamental ambiguity known as the shape-radiance ambiguity.
To an optimizer, a bright spot on an object could be a physical bump catching the light, a light-colored patch painted onto a flat surface, or a specular reflection of a distant light source. Without constraints, joint optimization algorithms often take the path of least resistance, incorrectly "baking" transient reflections, highlights, or shadows directly into the permanent 3D geometry. This results in noisy, distorted shapes and inaccurate material models.
To solve this, a powerful paradigm has emerged across inverse rendering and neural reconstruction: staged or progressive optimization. Instead of solving the entire problem at once, these methods break the optimization landscape into sequential phases, establishing a stable geometric foundation before introducing complex, view-dependent appearance properties.
Phase 1: Establishing a Geometric Foundation
The core reason staged optimization works is that macro-geometry can often be reliably approximated by simplifying the material model. In the early stages of training, methods intentionally suppress or ignore complex view-dependent effects like specular highlights, treating the world as largely diffuse or Lambertian. This drastically simplifies the optimization landscape, allowing the model to lock down the correct spatial structure without interference from misleading reflections.
This strategy is frequently implemented via loss scheduling or progressive component activation:
- Loss Annealing: In DeLiRa: Self-Supervised Depth, Light, and Radiance Fields, Vitor Guizilini applies an annealing schedule to photometric consistency losses. The model initially prioritizes Lambertian consistency to establish the correct scene structure before phasing out the constraint to capture complex, view-dependent reflections.
- Diffuse-to-Specular Progression: In 3iGS: Factorised Tensorial Illumination for 3D Gaussian Splatting, Tat-Jen Cham et al. employ a two-stage strategy that first establishes basic geometry and appearance through diffuse color optimization. Only when the base structure is stable does the model introduce specular components and illumination fields to capture high-frequency reflections.
- Coarse-to-Fine Sequences: For dynamic scenes where motion compounds the ambiguity, TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing by Jiachen Tao et al. and SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes by Cheng-De Fan et al. use coarse-to-fine training sequences to stabilize base geometry before introducing motion and specular effects. This prevents complex reflections from distorting the underlying surface structure.
Alternatively, methods can use explicit geometric regularization or masking to shield the initial geometry from specular corruption. For instance, Adaptive Shells for Efficient Neural Radiance Field Rendering by Tianchang Shen et al. enforces strict geometric regularization early on to define a stable surface topology. When dealing with explicit reflectors, Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting by Jiarui Meng et al. masks out highly reflective regions entirely during the initial phase, preventing the formation of artifactual geometry caused by light transport ambiguity.
Phase 2: Freezing Geometry to Refine Appearance
Once the macro-geometry is established, the optimization shifts focus to recovering fine-grained appearance details, material properties (such as spatially-varying BRDFs), and complex lighting environments. Crucially, to prevent high-frequency specular details or lighting gradients from corrupting the recovered shape, the geometric parameters are typically frozen or heavily constrained during this phase.
[ Phase 1: Optimize Geometry ] ──> ( Freeze Spatial Parameters ) ──> [ Phase 2: Optimize Reflectance/BRDF ]
(Diffuse/Lambertian) (Specular/View-Dependent)
This freezing strategy prevents the model from shifting surface boundaries to explain away complex light transport effects:
- Decoupled Reflection Modeling: In ReflFlow: Learning Geometry-Guided Ray Tracing for Dynamic Specular Reconstruction, Jiachen Tao et al. isolate reflection modeling by training it exclusively with frozen geometry. This prevents the model from incorrectly baking reflections into the static texture of the geometry before a final joint fine-tuning phase.
- Handling Transparency and Sparsity: In TSGS: Improving Gaussian Splatting for Transparent Surface Reconstruction via Normal and De-lighting Priors, Mingwei Li et al. freeze spatial parameters before optimizing for complex visual effects, preventing high-frequency specular details from distorting transparent surface boundaries. Similarly, the authors of Rendering Anywhere You See: Renderability Field-guided Gaussian Splatting freeze spatial structures in their second stage to safely refine spherical harmonic coefficients using high-quality source images.
- Cross-Modal Disentanglement: This decoupling is also vital when training on heterogeneous data. For example, EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting by Bohao Liao et al. optimizes structural geometry and brightness using event data in the first stage, then freezes those geometric parameters to focus exclusively on optimizing color attributes using RGB frames, ensuring that grayscale event data does not degrade color accuracy.
Multi-Stage Factorization and Alternating Optimization
While a two-stage (geometry-then-appearance) pipeline is highly effective, more complex inverse rendering tasks require finer factorization or iterative schedules to handle the interplay of light, shadows, and materials.
Multi-Phase Factorization
Instead of grouping all appearance attributes into a single second phase, several frameworks progressively introduce individual physical components one by one. This ensures that specific visual phenomena are not incorrectly attributed to the wrong physical term:
- Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects by Ningjing Fan et al. factorizes inverse rendering into a progressive three-stage pipeline: first geometry, then light visibility, and finally material properties and environment lighting.
- SSD-GS: Scattering and Shadow Decomposition for Relightable 3D Gaussian Splatting by Iris Zheng utilizes a progressive optimization schedule that introduces light-material interaction components sequentially, starting with diffuse lighting and moving through shadows, subsurface scattering, and finally specular effects.
Alternating Optimization
Rather than utilizing a strict linear sequence, some frameworks decouple parameters by alternating between them. This allows geometry and appearance to co-evolve without competing in the same optimization step:
┌──────────────────────────────┐
│ Optimize Surface Geometry │◄─┐
└──────────────┬───────────────┘ │
│ │ Alternating
▼ │ Iterations
┌──────────────────────────────┐ │
│ Optimize Material Properties │──┘
└──────────────────────────────┘
In Multiview Shape and Reflectance from Natural Illumination, Geoffrey Oxholm et al. introduce an iterative optimization strategy that alternates between estimating surface geometry and material reflectance properties. By fixing one set of parameters to refine the other, the system decouples their complex interdependencies.
Similarly, Shape from Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing by Purvi Goel et al. alternates between refining geometric vertex positions and updating spatially-varying BRDF parameters. This iterative decoupling ensures that as the mesh geometry evolves, it provides increasingly accurate surface normals for reasoning about light transport, which in turn allows for a cleaner estimation of specular roughness and diffuse albedo.
Key Foundational Work
Efficient photometric stereo on glossy surfaces with wide specular lobes (2008) — contributed 2 concept elements to this direction.
Multiview Shape and Reflectance from Natural Illumination (2014) — contributed 1 concept elements to this direction.
Intrinsic Face Image Decomposition with Human Face Priors (2014) — contributed 1 concept elements to this direction.