TL;DR
This paper introduces a novel deep learning approach for estimating the illuminant of an image without requiring ground truth illuminant data. Instead, it trains an illuminant estimation network by leveraging an auxiliary task: object recognition. This method offers a promising way to achieve accurate color constancy by learning from readily available object labels, potentially simplifying data acquisition for future research.
What Problem Does This Paper Address?
Color constancy is the ability to perceive colors consistently under varying lighting conditions. Computational color constancy aims to replicate this by estimating the scene's illuminant and then correcting the image. Traditional "learned" methods for this task rely heavily on large datasets with meticulously labeled illuminant information. However, acquiring such ground truth data is expensive and time-consuming. This paper addresses the challenge of training effective illuminant estimation models without this costly ground truth, drawing inspiration from how the human visual system might have evolved color constancy as a byproduct of other crucial tasks like object recognition.
Methods
The core of the proposed method is an end-to-end trained network called IEOR (Illuminant Estimation / Object Recognition). It consists of two main modules:
- Illuminant Estimation (IE) module: This module takes an input RGB image and predicts the scene illuminant, typically represented as a diagonal matrix (e.g.,
(R, G, B)values). - Object Recognition (OR) module: This module performs an auxiliary supervised task (in this case, vegetable recognition).
The key innovation lies in the training strategy. The IE module is trained indirectly by optimizing the performance of the OR module. During training, an input image is first processed by the IE module to estimate the illuminant and correct the image. This corrected image is then fed into the OR module. The loss signal from the OR module's classification task is backpropagated through both modules, but only the weights of the IE module are updated. This forces the IE module to learn to produce color corrections that are beneficial for object recognition, effectively learning illuminant estimation as a byproduct.
To encourage this separation of tasks, the training involves two phases:
- Pre-training the OR module alone on the chosen auxiliary task.
- Connecting IE and training the entire system end-to-end, with the addition of "color jittering" (artificially augmenting the input with random illuminants) to make the OR module more sensitive to color variations.
Datasets
- Training: The authors use a subset of the VegFru dataset for training. This dataset is chosen because color is a discriminating feature for vegetable recognition, making it suitable for the auxiliary task.
- Evaluation: The proposed method is evaluated on two standard color constancy benchmarks:
- Shi-Gehler dataset: Contains 568 images with ColorChecker targets for ground truth extraction.
- NUS dataset: Contains 1853 images from 9 different cameras, also using ColorChecker ground truth.
Key Insights and Ideas
- Learning without Ground Truth: The most significant insight is demonstrating that accurate illuminant estimation can be achieved by training a network for an auxiliary task (object recognition) without needing explicit illuminant annotations. This bypasses the costly data acquisition bottleneck.
- Illuminant Estimation as a Byproduct: The paper validates the hypothesis that color constancy abilities can emerge as a beneficial side effect of tasks that rely on consistent color perception, such as object recognition.
- Auxiliary Task Design: The choice of an auxiliary task where color is important (like vegetable recognition) is crucial for effectively driving the illuminant estimation module.
- Color Jittering's Role: Artificially introducing color variations during training helps to make the object recognition module more sensitive to color changes, thereby forcing the illuminant estimation module to learn more robust corrections.
Key Results
- The proposed IEOR method, when evaluated on standard datasets (Shi-Gehler and NUS), achieves competitive results.
- In cross-dataset evaluation (trained on VegFru, tested on Shi-Gehler/NUS), the method outperforms most other deep learning methods.
- The classification-based training strategy leads to up to a 19% improvement in angular error compared to training the illuminant estimation module directly for regression on the same augmented data.
- The global normalization variant of the method shows robust performance across both datasets.
Limitations and Future Work
- Training Set Color Bias: The authors acknowledge that the illuminant distribution in the classification training dataset can influence the learned illuminant. If the average illuminant in the training set is not neutral, the predicted illuminants on test sets might not match ground truth without further adjustments.
- Dataset Heterogeneity: High variability in the training dataset can make the object recognition module overly robust to color variations, potentially hindering the IE module's learning.
- Future Work: The authors plan to extend their approach to more complex illumination scenarios, such as those with multiple illuminants. They note that their end-to-end learning framework would not require ground truth data for these challenging scenarios, which are currently difficult to address due to the lack of large annotated datasets. They also suggest preprocessing the training set for unsupervised white balancing to reduce intra-class variability.
Summary updated