Deep neural models for color classification and color constancy

Alban Flachot, Arash Akbarinia, Heiko H. Schütt, Roland W. Fleming, Felix A. Wichmann, Karl R. GegenfurtnerJournal of Vision 2022 · DOI

TL;DR

This paper trains deep neural networks (DNNs) to perform color constancy, a crucial aspect of human vision. They demonstrate that DNNs can achieve supra-human levels of color constancy by learning to discount illumination variations, and importantly, that a specific network architecture (DeepCC) learns color representations that closely mirror human perception, suggesting these representations are key to robust color constancy.

What Problem Does This Paper Address?

Color constancy is our remarkable ability to perceive the true color of an object despite changes in the illumination. While humans are quite good at this, understanding the underlying neural mechanisms and computational strategies remains a challenge. Previous research has explored this through behavioral studies and simpler computational models, but a comprehensive neural model that captures both human-like performance and internal representations has been elusive. This paper aims to bridge this gap by training deep neural networks on a color constancy task and analyzing their performance and internal workings to see if they can mimic and potentially explain human color constancy.

Methods

The authors employed a deep convolutional neural network (CNN) architecture, specifically a simplified model they call DeepCC, for their experiments. They also tested more complex architectures like ResNets. The core idea was to train these networks to classify the surface reflectance of objects, which is the intrinsic color of the object, rather than just its perceived color under a specific illumination. This requires the network to learn to discount the effect of the illumination.

Key methodological aspects include:

  • Spectral Rendering: Using a physically-based renderer (Mitsuba) to create realistic 3D scenes with spectral data for both object reflectances and illuminations. This allows for precise control over the visual input.
  • LMS Cone Inputs: Converting the rendered multispectral images into three "LMS" channels, mimicking human cone photoreceptor responses, as input to the networks.
  • Supervised Learning: Training the networks using cross-entropy loss to predict the correct Munsell chip label for each object.
  • Evaluation Metrics: Employing standard metrics like top-1 and top-5 accuracy, alongside custom metrics like Muns³ accuracy (within 1 Munsell step), ΔE (Euclidean distance in CIELab* space), and the Color Constancy Index (CCI) to quantify performance and color constancy.
  • Representational Similarity Analysis (RSA): Using multidimensional scaling (MDS) to visualize and compare the internal color representations learned by different layers of the networks with human perceptual color spaces (Munsell and CIELab*).

Datasets

The authors generated their own datasets for training and evaluation:

  • Training Datasets:
    • Set-CC: Contained 279 natural illuminations and 1,600 Munsell chip reflectances, resulting in 446,400 images. This dataset was designed to require color constancy.
  • Set-D65: Similar to Set-CC but used a constant D65 illumination, serving as a baseline for tasks not requiring color constancy.
  • Evaluation Datasets:
    • Normal Condition: Used a subset of 330 Munsell chips and four specific illuminations (Yellow, Blue, Green, Red) to test performance under controlled conditions, including illuminations outside the training distribution.
  • Impoverished Visual Scenes: Created to test the reliance on contextual cues by manipulating the scene:
  • No patch: Removed colored patches from the background.
  • Wrong patch: Replaced background patches with their colors under a different illumination.
  • Wrong background: Placed the object in a scene illuminated differently from its background.
  • Comparison Datasets: Used standard approaches like Gray World, White Patch, and ASM, coupled with von Kries correction, for comparison.

Key Insights and Ideas

  • DNNs can achieve supra-human color constancy: The trained models, particularly DeepCC, performed exceptionally well, often exceeding human capabilities in classifying object colors under varying illuminations.
  • Context is crucial for color constancy: Manipulating contextual cues (like background patches) significantly degraded the performance of the DeepCC model, highlighting its reliance on scene context, similar to humans.
  • Daylight locus bias is learned: The models showed better color constancy for illuminations along the "daylight locus" compared to orthogonal directions, mirroring human biases and suggesting this is learned from the naturalistic training data.
  • Human-like color representations are not strictly necessary for performance, but are beneficial: While other architectures achieved high performance without learning human-like color representations, DeepCC's ability to do so suggests it might be a better model for understanding human color perception.
  • DeepCC learns a progressive disentanglement of color dimensions: As information flows through the network's layers, DeepCC increasingly separates color information into dimensions that align with human perception (hue, lightness, saturation).

Key Results

  • High Accuracy: DeepCC achieved median top-1 accuracies of 76% on the CC validation set and 86% on the D65 validation set. On the evaluation dataset, median top-1 accuracy was around 80% under normal conditions.
  • Color Constancy Index (CCI): DeepCC achieved a median CCI of 0.83 in 75% of cases, which is considered high and comparable to human performance.
  • Impact of Context Removal: In the "no patch" condition, DeepCC's accuracy dropped to a median of 40%, and in the "wrong background" condition, it dropped to a median top-1 accuracy of 0%, demonstrating a strong reliance on context.
  • Architectural Differences in Representation: While other architectures like ResCC performed well on the task, their internal color representations did not align with human perceptual dimensions, unlike DeepCC. DeepCC's first fully connected layer (fc1) explained 91% of the variance in its Munsell representations, closely matching human perception.

Limitations and Future Work

  • Limitations:
    • The study primarily used rendered scenes, and extending findings to real-world, complex outdoor scenes remains a challenge.
  • While DeepCC learned human-like representations, the exact neural mechanisms responsible for color constancy are still unclear and may not be directly comparable to biological systems.
  • The study focused on a specific classification task; color constancy might involve more complex processes in natural vision.
  • Future Work:
    • Investigating how different architectures learn color representations and their impact on performance.
  • Exploring the role of adaptation mechanisms in color constancy, potentially by incorporating them into DNN architectures.
  • Developing more sophisticated methods to analyze and understand the learned representations in DNNs.
  • Applying these findings to improve computer vision systems for tasks beyond simple color correction.

Summary updated

Abstract

Color constancy is our ability to perceive constant colors across varying illuminations. Here, we trained deep neural networks to be color constant and evaluated their performance with varying cues. Inputs to the networks consisted of two-dimensional images of simulated cone excitations derived from three-dimensional (3D) rendered scenes of 2,115 different 3D shapes, with spectral reflectances of 1,600 different Munsell chips, illuminated under 278 different natural illuminations. The models were trained to classify the reflectance of the objects. Testing was done with four new illuminations with equally spaced CIEL*a*b* chromaticities, two along the daylight locus and two orthogonal to it. High levels of color constancy were achieved with different deep neural networks, and constancy was higher along the daylight locus. When gradually removing cues from the scene, constancy decreased. Both ResNets and classical ConvNets of varying degrees of complexity performed well. However, DeepCC, our simplest sequential convolutional network, represented colors along the three color dimensions of human color vision, while ResNets showed a more complex representation.
BibTeX
@article{flachot2022deep,
  author = {Alban Flachot and Arash Akbarinia and Heiko H. Schütt and Roland W. Fleming and Felix A. Wichmann and Karl R. Gegenfurtner},
  title = {{Deep neural models for color classification and color constancy}},
  journal = {Journal of Vision},
  year = {2022},
  doi = {10.1167/jov.22.4.17},
}
Figure