When we look at a photograph of a shiny teapot, our brains effortlessly distinguish between its physical shape, the ceramic material it is made of, and the reflection of the kitchen window on its surface. For a computer, this is an "ill-posed" nightmare known as the shape-radiance ambiguity. Without constraints, a neural network might decide that a reflection is actually a bright spot painted onto the teapot, or that a shadow is a physical dip in the geometry.
To solve this, researchers have converged on a shared strategy: staged optimization. Instead of asking a model to learn everything at once, they break the problem into a sequence of dependencies. By establishing a stable geometric foundation before introducing complex appearance modeling, these methods prevent the "baking" of lighting and reflections into the static structure of the scene.
The Geometry First Mandate
The most common failure in joint optimization is when the model uses texture or lighting to compensate for poor geometry. If the 3D structure is slightly off, the model will simply "paint" the correct colors onto that incorrect shape to minimize the training loss.
To prevent this, frameworks like PSRGS by Bocheng Li and IRGS by Chun Gu employ a two-stage process. They first optimize for a coarse geometric representation—often using depth-aware losses—to lock in the physical boundaries of the scene. Only after the geometry is "frozen" or stabilized do they allow the model to refine high-frequency textures or complex material properties. This ensures that texture loss is not incorrectly absorbed by geometric parameters.
This principle extends even to specialized sensors. In EF-3DGS, Bohao Liao uses event cameras to capture high-speed motion. Because event data is grayscale and noisy, the model first optimizes structural geometry and brightness. Only in the second stage does it freeze those spatial parameters to focus exclusively on recovering accurate color from sparse RGB frames, preventing the noisy event data from corrupting the final color representation.
Resolving the Material Lighting Ambiguity
Once geometry is stable, the next challenge is disentangling the Bidirectional Reflectance Distribution Function (BRDF)—the "material"—from the incident light.
In PBR-NeRF, Sean F. Wu structures the optimization into phases: initialization of geometry, estimation of materials and lighting, and finally a joint refinement. By establishing the surface normals first, the physics-based rendering (PBR) constraints have a reliable surface to "bounce" light off of. Similarly, Bang Du in GlossGau locks the geometry to allow for the isolated refinement of reflectance parameters, specifically to prevent the "baking" of specular highlights into the base texture.
This staged approach is particularly critical for glossy surfaces. Jiachen Tao and the authors of TraceFlow use a coarse-to-fine strategy that optimizes diffuse geometry before introducing reflection logic. By the time the model starts calculating complex ray-traced reflections, the "mirror" (the surface geometry) is already perfectly placed, preventing the optimization from confusing a reflected object with a physical one behind the glass.
Stability Through Parameter Scheduling
Staging isn't always about freezing layers; sometimes it's about controlling the "speed" of learning for different attributes. In Time of the Flight of the Gaussians, Runfeng Li significantly reduces the learning rate for surface reflectivity relative to spatial parameters. This forces the optimization to resolve the 3D structure using the most reliable signals before it is allowed to "tweak" the reflectivity to match the appearance.
A similar logic applies to "in-the-wild" data. Moustafa Meshry in Neural Rerendering in the Wild uses a staged pipeline to decouple scene appearance from geometric structure. By training an appearance encoder separately from the rerendering network, the system learns to ignore transient elements like weather or crowds, focusing instead on the permanent features of a landmark.
Beyond Physics: Semantics and Avatars
The staged optimization paradigm has proven so robust that it is now used to disentangle non-physical attributes, such as semantic labels and human poses.
- Human Avatars: In SAGA, Yang Cong uses an "adhere-then-detach" strategy. The model first binds 3D primitives strictly to a human template mesh to ensure structural integrity, then relaxes these constraints in a second stage to capture fine details like flowing hair or clothing wrinkles.
- Semantic Understanding: Jie Zhang in Semantic-Human delays semantic optimization until the underlying geometry is stable. This ensures that noisy 2D labels don't pull the 3D geometry out of alignment during the early stages of training.
- Colorization: In ChromaDistill, Venkatesh Babu Radhakrishnan first optimizes a radiance field on luminance (grayscale) data alone. Only after the 3D structure is established does the model learn to "hallucinate" color through distillation, preventing geometric inaccuracies from corrupting the color representation.
Why It Works: Reducing the Search Space
The success of staged optimization across these diverse papers—from SPARF's camera pose refinement to NC-SDF's indoor reconstruction—stems from a simple mathematical reality: search space reduction.
By solving for geometry first, the model effectively removes millions of "wrong" solutions where lighting or materials could have been used to mask a bad shape. As Ziyu Chen demonstrates in L-Tracing, once you have a fixed surface, calculating light visibility becomes a deterministic geometric problem rather than a vague optimization guess. Staging transforms a chaotic, multi-dimensional search into a structured, manageable pipeline.