Color Constancy based on Image Similarity via Bilayer Sparse Coding

Bing Li, Weihua Xiong, Weiming Hu

TL;DR

This paper introduces a novel "bilayer sparse coding" model for color constancy that simultaneously considers both low-level color information and high-level scene content. By integrating these two aspects within a unified framework, the proposed method achieves superior illumination estimation accuracy compared to existing approaches, including those that combine multiple methods.

What Problem Does This Paper Address?

Color constancy is a fundamental problem in computer vision aiming to make images appear as if they were captured under a standard illumination, regardless of the actual lighting conditions. The color of an object's surface is influenced by three factors: the light source, the surface's reflectance, and the camera's sensor. Variations in illumination can drastically alter how colors are perceived. While humans possess innate color constancy, achieving this computationally is challenging because estimating the illumination color is an ill-posed problem. Early methods treated images as simple collections of pixels, but recent research has shown the benefit of incorporating high-level visual content. However, these "content-driven" methods often rely on complex, multi-step processes like classifying scenes into predefined categories and then selecting or combining illumination estimates. This paper aims to bridge this gap by proposing a more integrated approach that directly leverages image similarity in both color and scene content for more robust and accurate illumination estimation.

Methods

The core contribution is a Bilayer Sparse Coding (BSC) model. This model operates on two "layers":

  • Color Layer: This layer uses sparse coding to reconstruct the test image's color feature (represented by a binarized 3D chromaticity histogram) using the color features of training images. This leverages the hypothesis that images with similar color distributions are likely captured under similar lighting. The reconstruction is guided by the similarity of color gamuts.
  • Scene Layer: This layer also employs sparse coding, but this time to reconstruct the test image's scene content (represented by visual vocabulary histograms derived from SIFT descriptors and a Bag-of-Words model) using the scene content of training images. This is based on the idea that scenes belonging to the same high-level category often share similar illumination conditions. A multi-task joint sparse representation is used here to account for object co-occurrence within a scene.

Crucially, these two layers are mutually constrained. The model is optimized iteratively, where the color layer's optimization influences the scene layer's, and vice-versa. This integrated approach avoids the need for explicit scene classification, which can be error-prone. The final illumination estimation is then derived as a weighted average of the training images' illumination values, with weights determined by the learned sparse coefficients from both layers.

Datasets

The paper evaluates its method on two real-world image datasets:

  • Gehler-Shi Set: This dataset consists of 568 images captured by high-quality DSLR cameras, including a variety of indoor and outdoor scenes. The data was reprocessed to provide linear RGB images.
  • SFU Set: This dataset comprises over 11,000 frames from videos, with a subset of 1,135 images selected to reduce redundancy. This dataset is notable for having ground truth illumination information derived from a grey sphere ball.

Key Insights and Ideas

  • Unified Bilayer Framework: The most significant insight is the successful integration of low-level color and high-level scene content into a single, mutually constrained sparse coding framework. This avoids the limitations of separate, sequential processing steps.
  • Implicit Scene Understanding: By using sparse coding for scene content similarity, the model implicitly captures scene category information without requiring explicit, potentially error-prone, classification. This is a key advantage over previous content-driven methods.
  • Image Similarity as a Core Principle: The paper strongly emphasizes that image similarity, in both color and scene, is a powerful cue for illumination estimation. This principle is directly encoded into the sparse coding formulations.
  • Sparse Coding's Versatility: The paper demonstrates the effectiveness of sparse coding not only for color feature reconstruction but also for scene content representation and similarity analysis.

Key Results

The proposed Bilayer Sparse Coding (BSC) method consistently outperformed existing methods on both the Gehler-Shi and SFU datasets.

  • Gehler-Shi Set: BSC achieved the best median and trimean errors, outperforming even leading "content-driven" combinational methods like IO. For instance, BSC with m=800 achieved median/trimean errors of 2.06/2.71, compared to IO's 2.23/2.83. The single-layer sparse coding (SSC) method performed comparably to SVR, highlighting the effectiveness of sparse coding itself, but BSC's bilayer approach provided further significant improvements.
  • SFU Set: Similar results were observed, with BSC consistently outperforming other methods, including IO. BSC with m=800 achieved median/trimean errors of 3.95/4.31, while IO had 4.22/4.96. The paper also provides visual examples (Figure 2) demonstrating how BSC's use of scene content leads to more accurate illumination estimation by selecting more relevant training images compared to SSC.

Limitations and Future Work

  • Parameter Tuning: The BSC model requires setting regularization coefficients (λ, β) and the number of visual words (m) for the Bag-of-Words model. While the paper suggests fixed values for λ and β and explores different values for m, optimal parameter selection can still be a challenge.
  • Computational Cost: Sparse coding and multi-task joint sparse representation can be computationally intensive, although the paper proposes an efficient iterative optimization algorithm.
  • Future Work: The paper implicitly suggests that further exploration of different feature representations for both color and scene content could lead to even better performance. Additionally, investigating more sophisticated ways to integrate the two layers or exploring other types of scene content cues could be fruitful avenues for future research.

Summary updated

Abstract

Computational color constancy is a very important topic in computer vision and has attracted many researchers' attention. Recently, lots of research has shown the effects of high level visual content information for illumination estimation. However, all of these existing methods are essentially combinational strategies in which image's content analysis is only used to guide the combination or selection from a variety of individual illumination estimation methods. In this paper, we propose a novel bilayer sparse coding model for illumination estimation that considers image similarity in terms of both low level color distribution and high level image scene content simultaneously. For the purpose, the image's scene content information is integrated with its color distribution to obtain optimal illumination estimation model. The experimental results on two real-world image sets show that our algorithm is superior to other prevailing illumination estimation methods, even better than combinational methods.
BibTeX
@misc{li2012color,
  author = {Bing Li and Weihua Xiong and Weiming Hu},
  title = {{Color Constancy based on Image Similarity via Bilayer Sparse Coding}},
  year = {2012},
  eprint = {1207.3142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1207.3142},
}
Figure