Modeling View Dependent Appearance on Non Lambertian Surfaces

184 concepts150 papers1 proposal

When we look at a matte brick wall, its color and texture remain virtually identical regardless of our viewing angle. This property is known as Lambertian reflectance. For decades, computer vision and 3D reconstruction algorithms relied heavily on the Lambertian assumption: the belief that a physical point in space possesses a single, constant color.

However, the real world is filled with non-Lambertian surfaces. Consider a polished chrome sphere, a glass bottle, or a glossy apple. As you move around a glossy apple, the bright white specular highlight shifts across its surface. The highlight is not painted onto the object; it is a reflection of an external light source whose observed position depends entirely on the relationship between the light, the surface normal, and the camera.

If a reconstruction algorithm forces a single static color onto each 3D coordinate, it encounters a fundamental paradox. To reconcile why a point looks white from one angle and red from another, the algorithm must either deform the geometry (creating fake holes or floating artifacts) or blur the appearance into a dull, matte average. Overcoming this challenge requires explicitly modeling how appearance changes as a function of the viewing angle.

The Shape-Radiance Ambiguity

When algorithms attempt to reconstruct a scene from sparse viewpoints, they must solve a chicken-and-egg problem: is a change in pixel intensity caused by a change in the underlying geometry, or is it a view-dependent reflection? This is known as the shape-radiance ambiguity.

As argued by Wang et al. in StructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints, enforcing strict multi-view consistency can inadvertently suppress the very features that make a material look realistic. When an optimization routine assumes a static appearance across viewpoints, it treats specular highlights as errors. To minimize these errors, the system may flatten out the highlights or corrupt the geometry. Kwak et al. in GeCoNeRF: Few-shot Neural Radiance Fields via Geometric Consistency similarly note that rigid pixel-to-pixel matching fails on non-Lambertian surfaces, necessitating representations that decouple geometry from perspective-shifted radiance.

Conditioning Radiance on Direction

The breakthrough of modern neural rendering frameworks lies in how they parameterize light. Instead of representing a scene as a 3D grid of static colors, frameworks like the one introduced by Mildenhall et al. in NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis model the scene as a continuous 5D function.

Traditional 3D Voxel:  (x, y, z) ---------> Color
Neural Radiance Field: (x, y, z, θ, ϕ) ----> Color, Density

By feeding both the 3D spatial coordinates $(x, y, z)$ and the 2D viewing angles $(\theta, \phi)$ into a neural network, the model learns to output a different color for the exact same point in space depending on the direction from which it is observed. When rendering a glossy surface, the network uses the spatial coordinates to determine the surface's structural presence (density) and leverages the directional inputs to interpolate the shifting position of a specular glint.

Decomposing Transmitted and Reflected Fields

While a 5D coordinate mapping works well for smooth highlights, it struggles with perfect mirrors or transparent glass. On a glass window, the camera captures two distinct layers of light simultaneously: the transmitted light passing through the glass from the background, and the reflected light bouncing off the surface from the foreground environment. Furthermore, the ratio between these two layers changes dynamically based on the viewing angle—a physical phenomenon known as the Fresnel effect.

To capture this complexity without producing "ghostly" geometric artifacts, Guo et al. in NeRFReN: Neural Radiance Fields with Reflections proposed decomposing the scene into separate neural fields. By isolating the transmitted components from the reflected components, the model can accurately simulate how light interacts with polished or transparent barriers, preserving sharp environmental reflections without compromising the depth estimation of the underlying geometry.

Turning Reflections into Geometric Signals

Rather than treating view-dependent variations as noise to be smoothed away or bypassed, alternative approaches exploit them as a rich source of geometric information.

Nehab et al. demonstrated in Dense 3D reconstruction from specularity consistency that the predictable way a specularity shifts across multiple camera views provides a strict mathematical constraint. Because the movement of a highlight is governed by the curvature of the surface, tracking this "specular flow" allows for the precise triangulation of surface normals on mirror-like objects where traditional stereo matching completely fails.

Grounding Consistency in Polarization

Another way to bypass the failures of pixel-level visual consistency on shiny surfaces is to look at the physical properties of light waves. When unpolarized light strikes a non-Lambertian surface, the reflected light becomes partially polarized based on the orientation of the surface.

Unpolarized Light ---> [ Glossy Object Surface ] ---> Polarized Reflection
                                                              |
                                                    Reveals Surface Normal

Fukao leveraged this principle in Polarimetric Normal Stereo to directly recover high-resolution surface normals for glossy and metallic objects. By integrating polarimetric cues into the matching process, the system grounds its optimization in a physical invariant rather than shifting pixel intensities.

Taking this a step further into self-supervised learning, Ruhkamp et al. introduced S2P3: Self-Supervised Polarimetric Pose Prediction. The authors utilized an invertible physical model to link predicted object poses back to raw polarization data. This ensures that the model learns from the invariant physical properties of light transport, enabling robust tracking and pose estimation for photometrically challenging objects like glass bottles and metal cutlery.

View-Dependency in Explicit and Generative Architectures

While implicit continuous functions (like NeRFs) handle directional inputs naturally through multi-layer perceptrons, explicit representations like Multiplane Images (MPIs) or voxel grids must find alternative ways to inject view-dependency without exploding their memory footprint.

In the generative domain, Kumar et al. addressed this in Generative Multiplane Neural Radiance for 3D-Aware Image Generation by incorporating a view-dependent network layer directly into an MPI framework. This prevents the "flat" or matte appearance typical of standard explicit layers, allowing synthesized images to exhibit dynamic lighting changes during camera rotation.

Standard MPI:       [Layer 1 (Static)] -> [Layer 2 (Static)] -> Flat Rendering
View-Dependent MPI: [Layer 1] + [Directional Embedding] --------> Dynamic Reflections

Alternatively, Cham reimagined the rendering pipeline entirely in ABLE-NeRF: Attention-Based Rendering with Learnable Embeddings for Neural Radiance Field. Instead of relying on physics-based volumetric ray-marching, Cham replaced the core pipeline with a transformer-based attention mechanism. Here, "Learnable Embeddings" act as virtual light probes distributed throughout the space. By using attention to query these embeddings based on the viewing ray, the model learns to synthesize complex, multi-bounce reflections and translucency, bypassing traditional light transport modeling entirely to achieve high-fidelity view-dependent synthesis.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_view_dependent_appearance_modeling_c688d0dc,
  title = {view-dependent appearance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/view-dependent-appearance-modeling/rdr_c688d0dc7f0d42a491e5406de348b866},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_view_dependent_appearance_modeling_c688d0dc_v1,
  title = {view-dependent appearance modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/view-dependent-appearance-modeling/rdr_c688d0dc7f0d42a491e5406de348b866/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake