Deep neural networks trained for estimating reflectance and illumination achieve lightness constancy differently than human observers

Alban Flachot, Jaykishan Patel, Thomas Sa Wallis, Marcus A. Brubaker, David H. Brainard, Richard F. MurrayJournal of Vision 2026 · DOI

Abstract

Lightness constancy, the ability to create perceptual representations that are strongly correlated with surface albedo despite variations in lighting and context, is a challenging computational problem. Indeed, it has proven difficult to develop image-computable models of how human vision achieves a substantial degree of lightness constancy in complex scenes. Recently, convolutional neural networks (CNNs) have been developed that are proficient at estimating albedo, but little is known about how they achieve this, or whether they are good models of human vision. We examined this question by training a CNN to estimate albedo and illumination in a computer-rendered virtual world, and evaluating both the CNN and human observers in a lightness matching task. In several conditions, we eliminated cues potentially supporting lightness constancy: local contrast, shading, shadows, and all contextual cues. We found that the network achieved a high degree of lightness constancy, outperforming three classic models, and substantially outperforming human observers as well. However, we also found that eliminating cues affected the CNN and humans very differently. Humans had much worse constancy when local contrast cues were made uninformative, but were minimally affected by elimination of shading or shadows. The CNN was unaffected by local contrast, but relied on shading and shadows. These results suggest that the CNN followed an effective strategy of integrating global image cues, whereas humans used a more local strategy. In a follow-up experiment, we found that the CNN could learn to exploit noise artifacts that were correlated with illuminance in ray-traced scenes, whereas humans did not. We conclude that CNNs can learn an effective, global strategy of estimating lightness, which is closer to an optimal strategy for the ensemble of scenes we studied than the computation used by human vision.

Introduction

A fundamental mystery of vision is lightness constancy: our ability to perceive a white piece of paper as "white" whether it is under a dim desk lamp or bright sunlight. While modern Deep Neural Networks (DNNs) have become incredibly proficient at this task, a critical question remains: do these "super-human" AI models actually see the world the way we do? This study explores the gap between image-computable models and human perception by systematically "breaking" both observers to identify which visual cues are essential for each.

Methods

The researchers devised a lightness matching task using computer-rendered 3D scenes. Observers (9 humans and a custom CNN) matched the albedo (reflectance) of a "match patch" to a "reference patch" under varying lighting conditions.

Experimental setup and cue manipulation. (A) Example scene and its RGB decomposition. (B) Matrix of lighting conditions and cue manipulations, including removal of local contrast, shading, and shadows.

Performance was measured using the Thouless ratio, where 0 indicates matching raw pixel brightness (no constancy) and 1 indicates perfect perception of true surface color. The CNN used a modified U-Net architecture with dual decoders for albedo and illumination, trained with a reconstruction loss to obey the physics of $Luminance = Albedo \times Illumination$.

The experiment featured six conditions created by selectively removing cues:

  1. Local Contrast: Making the area around the patches uninformative.
  2. Shading: Removing light gradients on objects.
  3. Shadows: Removing cast shadows.
  4. Shading & Shadows: Removing both.
  5. No Cue: Removing all contextual information.

Results

The CNN achieved a Thouless ratio of 0.79, nearly double the human average of 0.38. However, the two observers relied on fundamentally different strategies:

  • Global vs. Local Strategies: Humans are primarily local observers. When local contrast between an object and its immediate background was silenced, human constancy collapsed (dropping from ~0.32 to 0.09). In contrast, the CNN was unaffected by local contrast, instead integrating global cues like shading and shadows across the entire scene to calculate illumination.
  • CNN Vulnerability: The CNN's performance dropped significantly when shading and shadows were removed—cues that humans largely ignored in this task.
  • Opportunistic AI: In a follow-up experiment using ray-traced (Cycles) images, the CNN "cheated" by exploiting rendering noise. It learned that noisier pixels correlated with shadows, achieving a Thouless ratio of 0.07 in the "No-cue" condition where constancy should be impossible. Humans were unable to use this artifact.

Discussion

The superior performance of the CNN actually makes it a worse model of human vision in this context. It relies on global features that the human visual system ignores. This suggests that to make DNNs more "human-like," researchers might need to limit model capacity, employ multi-task training to force more robust local strategies, or move toward unsupervised learning paradigms that better mimic biological development.

Summary updated

Abstract

Lightness constancy, the ability to create perceptual representations that are strongly correlated with surface reflectance despite variations in lighting and context, is a challenging computational problem. Indeed, it has proven difficult to develop image-computable models of how human vision achieves a substantial degree of lightness constancy in complex scenes. Recently, convolutional neural networks have been developed that are proficient at estimating reflectance, but little is known about how they achieve this, or whether they are good models of human vision. We examined this question by training a convolutional neural network to estimate reflectance and illumination in a computer-rendered virtual world, and evaluating both the convolutional neural network and human observers in a lightness matching task. In several conditions, we eliminated cues potentially supporting lightness constancy: local contrast, shading, shadows, and all contextual cues. We found that the network achieved a high degree of lightness constancy, outperforming human observers. However, we also found that eliminating cues affected the convolutional neural network and humans very differently. Humans were most affected when local contrast cues were made uninformative, whereas the convolutional neural network mostly relied on shading and shadows. In a follow-up experiment, we found that the convolutional neural network could learn to exploit noise artifacts typically associated with ray tracing and correlated with illuminance, with potential implications for the many studies relying on ray-traced images. We conclude that convolutional neural networks can learn an effective, global strategy of estimating lightness, which is closer to an optimal strategy for the ensemble of scenes we studied than the computation used by human vision.
BibTeX
@article{flachot2026deep,
  author = {Alban Flachot and Jaykishan Patel and Thomas SA Wallis and Marcus A. Brubaker and David H. Brainard and Richard F. Murray},
  title = {{Deep neural networks trained for estimating reflectance and illumination achieve lightness constancy differently than human observers}},
  journal = {Journal of Vision},
  year = {2026},
  doi = {10.1167/jov.26.2.11},
}
Figure