Abstract
Lightness constancy, the ability to create perceptual representations that are strongly correlated with surface albedo despite variations in lighting and context, is a challenging computational problem. Indeed, it has proven difficult to develop image-computable models of how human vision achieves a substantial degree of lightness constancy in complex scenes. Recently, convolutional neural networks (CNNs) have been developed that are proficient at estimating albedo, but little is known about how they achieve this, or whether they are good models of human vision. We examined this question by training a CNN to estimate albedo and illumination in a computer-rendered virtual world, and evaluating both the CNN and human observers in a lightness matching task. In several conditions, we eliminated cues potentially supporting lightness constancy: local contrast, shading, shadows, and all contextual cues. We found that the network achieved a high degree of lightness constancy, outperforming three classic models, and substantially outperforming human observers as well. However, we also found that eliminating cues affected the CNN and humans very differently. Humans had much worse constancy when local contrast cues were made uninformative, but were minimally affected by elimination of shading or shadows. The CNN was unaffected by local contrast, but relied on shading and shadows. These results suggest that the CNN followed an effective strategy of integrating global image cues, whereas humans used a more local strategy. In a follow-up experiment, we found that the CNN could learn to exploit noise artifacts that were correlated with illuminance in ray-traced scenes, whereas humans did not. We conclude that CNNs can learn an effective, global strategy of estimating lightness, which is closer to an optimal strategy for the ensemble of scenes we studied than the computation used by human vision.
Introduction
A fundamental mystery of vision is lightness constancy: our ability to perceive a white piece of paper as "white" whether it is under a dim desk lamp or bright sunlight. While modern Deep Neural Networks (DNNs) have become incredibly proficient at this task, a critical question remains: do these "super-human" AI models actually see the world the way we do? This study explores the gap between image-computable models and human perception by systematically "breaking" both observers to identify which visual cues are essential for each.
Methods
The researchers devised a lightness matching task using computer-rendered 3D scenes. Observers (9 humans and a custom CNN) matched the albedo (reflectance) of a "match patch" to a "reference patch" under varying lighting conditions.
Performance was measured using the Thouless ratio, where 0 indicates matching raw pixel brightness (no constancy) and 1 indicates perfect perception of true surface color. The CNN used a modified U-Net architecture with dual decoders for albedo and illumination, trained with a reconstruction loss to obey the physics of $Luminance = Albedo \times Illumination$.
The experiment featured six conditions created by selectively removing cues:
- Local Contrast: Making the area around the patches uninformative.
- Shading: Removing light gradients on objects.
- Shadows: Removing cast shadows.
- Shading & Shadows: Removing both.
- No Cue: Removing all contextual information.
Results
The CNN achieved a Thouless ratio of 0.79, nearly double the human average of 0.38. However, the two observers relied on fundamentally different strategies:
- Global vs. Local Strategies: Humans are primarily local observers. When local contrast between an object and its immediate background was silenced, human constancy collapsed (dropping from ~0.32 to 0.09). In contrast, the CNN was unaffected by local contrast, instead integrating global cues like shading and shadows across the entire scene to calculate illumination.
- CNN Vulnerability: The CNN's performance dropped significantly when shading and shadows were removed—cues that humans largely ignored in this task.
- Opportunistic AI: In a follow-up experiment using ray-traced (Cycles) images, the CNN "cheated" by exploiting rendering noise. It learned that noisier pixels correlated with shadows, achieving a Thouless ratio of 0.07 in the "No-cue" condition where constancy should be impossible. Humans were unable to use this artifact.
Discussion
The superior performance of the CNN actually makes it a worse model of human vision in this context. It relies on global features that the human visual system ignores. This suggests that to make DNNs more "human-like," researchers might need to limit model capacity, employ multi-task training to force more robust local strategies, or move toward unsupervised learning paradigms that better mimic biological development.
Summary updated