Standard neural reconstruction methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have mastered the art of "faking" appearance. By optimizing for color consistency across views, they create stunningly photorealistic fly-throughs. However, these models often treat the world as a cloud of semi-transparent fog or a collection of unstructured points.
This "volumetric" shortcut breaks down the moment we want to change the lighting or render a shiny surface. To simulate how light bounces off a mirror or how a shadow falls across a face, a model needs to know exactly where the surface is and, more importantly, which way it is facing. A growing body of research is focused on recovering explicit surface normals within neural representations to bridge the gap between neural radiance and physically based rendering (PBR).
The Geometry of Reflection
The fundamental problem with volumetric primitives is that they lack a "front" and a "back." In standard 3DGS, for instance, the scene is composed of 3D ellipsoids. As Qi Zhang and Hongdong Li point out in their respective works on inverse rendering, these primitives do not inherently possess the surface normal information required for the rendering equation.
Why does this matter? Consider a specular reflection. The direction of a reflected ray is determined by the incident light and the surface normal ($R = V - 2(V \cdot N)N$). Because the reflection vector is so sensitive to the orientation of the surface, even a tiny error in the normal $N$ results in a massive displacement of the reflected image. Jiachen Tao and Haoxuan Wang demonstrate in TraceFlow that without precise normals, reflections of moving objects appear warped, blurry, or physically impossible.
From Clouds to Surfels
To solve this, researchers are moving away from 3D volumes toward 2D oriented primitives, often called "surfels." By constraining Gaussians to be thin, disk-like structures, researchers can treat the shortest axis of the primitive as an explicit surface normal.
Chun Gu and Li Zhang utilize this in IRGS, introducing a differentiable 2D ray tracer that calculates precise ray-surface intersections. This allows the model to simulate secondary light bounces—light hitting one surface and reflecting onto another—which is impossible in a purely volumetric representation. Similarly, Yuanhu Yao in Reflective Gaussian Splatting uses these oriented primitives to enable deferred rendering, a staple of modern game engines, within a neural framework.
Disentangling Light from Matter
The ultimate goal of recovering accurate normals is inverse rendering: decomposing a scene into its constituent parts—geometry, material properties (like roughness and metalness), and environment lighting.
When a model lacks geometric priors, it often "cheats" by baking lighting effects directly into the surface color. For example, a shiny highlight on a teapot might be reconstructed as a bright white spot on the texture rather than a reflection of a light source. Sean F. Wu and Tim Broedermann address this in PBR-NeRF by enforcing physics-based constraints. By requiring the model to obey the laws of energy conservation, they force the neural field to separate view-dependent reflections from the underlying diffuse color.
This decomposition is what enables "plug-and-play" relighting. Qi Zhang shows in GS-IR that once you have high-quality normals and materials, you can swap the environment map of a reconstructed scene to see how it would look at sunset or under neon lights, all in real-time.
The Challenge of "Fuzzy" Geometry
While oriented primitives work well for smooth surfaces like cars or furniture, they struggle with complex, non-manifold geometry like hair, fur, or fabric. These "fuzzy" objects don't have a single, well-defined surface normal.
Fujun Luan tackles this in RNG: Relightable Neural Gaussians by combining a neural reflectance decoder with specialized shadow cues. Instead of forcing a single normal onto a cluster of fur, the model learns a distribution of orientations. This allows for the relighting of soft, complex textures that traditional PBR pipelines—which expect clean meshes—typically fail to handle.
Why It Works
The shift toward surface-aware neural rendering works because it aligns the optimization process with the physics of the real world.
- Geometric Consistency: By forcing primitives to align into surfaces, models like GaussianShader by Yingwenqi Jiang reduce the "floater" artifacts common in early NeRFs.
- Radiometric Consistency: Kyu Beom Han demonstrates in RadioGS that minimizing the gap between a neural prediction and a physically-based render forces the geometry to be more accurate.
- Angular Sensitivity: As noted by Michael Guthe in early work on NURBS rendering, human vision is far more sensitive to errors in surface orientation (which affect shading and reflections) than to errors in absolute position. By prioritizing the normal, these new neural methods achieve a level of visual fidelity that "geometry-only" models cannot match.