Self-Supervised Inverse Rendering via Reflectance Consistency

36 concepts26 papers1 proposal

An image is a complex composite of physical reality. When we look at a photograph, our brains instantly separate the intrinsic properties of an object—its shape and material color—from the extrinsic properties of the environment, such as the position of the sun or the presence of shadows. For computer vision models, however, separating an image into its reflectance (albedo) and illumination components is a classic, highly ill-posed problem known as intrinsic image decomposition.

The fundamental challenge is a lack of ground truth. We cannot easily measure the true, unlit albedo of every object in a real-world scene to create a training dataset. To train deep networks without ground-truth labels, researchers leverage a simple but powerful physical prior: reflectance consistency. While illumination changes dramatically across different times of day, exposures, or viewpoints, the underlying material color of a physical point remains completely invariant.

The Siamese Architecture: Learning from Paired Illumination

The cleanest way to exploit this invariance is by observing the exact same scene under two different lighting conditions. If we feed both images into a weight-shared Siamese network, we can enforce a hard constraint: the predicted reflectance maps for both images must be identical.

This simple constraint completely changes the optimization landscape. Because the network is penalized if its two reflectance outputs differ, it cannot cheat by hiding lighting variations inside the reflectance channel. Instead, it is forced to attribute all pixel differences to the illumination channel.

Image A (Low Light)  ───► [ Weight-Shared ] ───► Reflectance A ──┐
                          [    Network    ]                      ├──► Loss: Force Equal
Image B (Bright Light) ──► [  (Siamese)   ] ───► Reflectance B ──┘

This strategy was pioneered by Wei-Chiu Ma et al. in Single Image Intrinsic Decomposition Without a Single Intrinsic Image, demonstrating that paired images of a scene under varying illumination provide a sufficient self-supervised signal for intrinsic decomposition. Similarly, Chen Wei et al. applied this concept to low-light enhancement in Deep Retinex Decomposition for Low-Light Enhancement, forcing a shared reflectance map between low-light and normal-light pairs to learn decomposition without explicit labels.

This paired consistency scales naturally across different exposure levels. For instance, Zhenqi Fu et al. introduced Learning a Simple Low-Light Image Enhancer from Paired Low-Light Instances, which penalizes differences in estimated reflectance maps across varying exposures to extract illumination-invariant features directly from data. In multi-exposure fusion, the authors of Retinex-MEF: Retinex-based Glare Effects Aware Unsupervised Multi-Exposure Image Fusion enforce a single shared reflectance component across all input exposures, preventing the visual artifacts that typically arise from independent estimations. This paired-consistency learning allows models to ignore transient illumination artifacts entirely, as shown by Francesco Nex et al. in Towards Learning Low-Light Indoor Semantic Segmentation with Illumination-Invariant Features.

Moving Beyond Static Viewpoints

What happens when the camera moves? If the viewpoint changes between exposures, pixels no longer align perfectly, and a simple pixel-to-pixel consistency loss fails. To resolve this, models must incorporate geometric understanding to map physical points across different views.

In InverseRenderNet: Learning Single Image Inverse Rendering, Ye Yu et al. utilized camera poses from Multiview Stereo (MVS) to cross-project pixels between views taken at different times. By enforcing photometric invariance across these warped views, the network learns to disentangle intrinsic material properties from transient outdoor illumination.

When dealing with view-dependent effects like specular highlights or reflections, the consistency constraint must be carefully isolated. Justus Thies addressed this in Image-guided Neural Object Rendering by enforcing that different source views, once stripped of their predicted view-dependent specularities, must project to an identical diffuse representation in a common coordinate space. For neural implicit surfaces, Lixin Yang demonstrated in Color-NeuS: Reconstructing Neural Implicit Surfaces with Color that applying a zero-mean constraint to the view-dependent relighting network forces the underlying implicit color network to capture only the true, static surface appearance.

Domain Applications: Shadows, Faces, and Physical Laws

The principle of reflectance consistency has been adapted to solve highly specific downstream tasks, such as shadow removal and face relighting, by introducing specialized structural and physical constraints.

Shadow Removal

Shadows represent a sharp spatial variation in illumination over an otherwise uniform material. To remove shadows without ground truth, Yuhao LIU et al. in Recasting Regional Lighting for Shadow Removal enforced spatially-variant priors requiring material color to remain consistent across shadow boundaries.

Similarly, Qing Zhang et al. developed a joint learning framework in When Shadow Removal Meets Intrinsic Image Decomposition: A Joint Learning Framework Using Unpaired Data, enforcing reflectance consistency between synthetically shadowed and shadow-free versions of the same scene. To generate reliable self-supervision signals from single images, Shilin Hu et al. proposed Shadow Removal Refinement via Material-Consistent Shadow Edges, fine-tuning foundation models to recognize continuous material regions regardless of lighting changes or shadow presence.

Face Relighting

Human faces possess complex geometry and skin reflectance. By training on pairs of images of the same subject under different lighting, Meng Wang et al. demonstrated in Face Inverse Rendering via Hierarchical Decoupling that the model is constrained to predict consistent albedo and geometry across both images due to the subject's constant identity. This self-supervised approach was also utilized by Youjia Wang et al. in Free-view Face Relighting Using a Hybrid Parametric Neural Model on a SMALL-OLAT Dataset to separate intrinsic albedo from extrinsic illumination using uncalibrated image sets.

Higher-Order Physical Constraints

Beyond simple identity matching, models can enforce higher-order physical laws. Yu-Ying Yeh et al. leveraged the principle of lighting linearity in Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation, enforcing a distributive consistency loss where the model's output for a combined light source must equal the sum of its outputs for individual light sources.

For domain adaptation, Zhipeng Du et al. introduced a circular consistency mechanism in Boosting Object Detection with Zero-Shot Day-Night Domain Adaptation, swapping and redecomposing estimated reflectance and illumination components to ensure the model extracts stable, lighting-independent features.

Key Foundational Work

Single Image Intrinsic Decomposition Without a Single Intrinsic Image (2018) — contributed 2 concept elements to this direction.
Deep Retinex Decomposition for Low-Light Enhancement (2018) — contributed 2 concept elements to this direction.
InverseRenderNet: Learning Single Image Inverse Rendering (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_exposure_reflectance_consistency_0f719ec6,
  title = {cross-exposure reflectance consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-exposure-reflectance-consistency/rdr_0f719ec6874247ac9d96efd20acdbdb2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_exposure_reflectance_consistency_0f719ec6_v1,
  title = {cross-exposure reflectance consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-exposure-reflectance-consistency/rdr_0f719ec6874247ac9d96efd20acdbdb2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake