Color constancy by combining low-mid-high level image cues

Yang Liu, Theo Gevers, Xueqing LiCVIU 2015 · DOI

TL;DR

This paper introduces a novel Bayesian framework for color constancy that integrates low-level image statistics, mid-level region information, and high-level scene context. By combining these diverse cues, the proposed method significantly outperforms existing state-of-the-art algorithms, achieving improved accuracy in estimating illuminant colors across various image types.

What Problem Does This Paper Address?

Color constancy is the ability to perceive the color of an object as consistent, regardless of the color of the light source illuminating it. Computationally, this is a challenging problem because the light source's color directly influences the colors captured by a camera. Traditional methods often rely on single, low-level image cues like pixel values or edges. While some approaches have combined multiple low-level cues, this paper addresses the gap of integrating a broader spectrum of information – from basic image statistics to semantic scene understanding – into a unified framework for more robust color constancy. The authors argue that high-level scene context (e.g., indoor vs. outdoor) can provide strong priors about potential light sources, which has been underexplored.

Methods

The core of the proposed method is a Bayesian framework. This framework allows for the principled combination of different "cues" (types of image information) by modeling their posterior probability for the illuminant. The paper breaks down these cues into three levels:

  • Low-level features: These include pixel values, and importantly, higher-order image statistics derived from the "n-jet" (first and second-order derivatives of image intensity). These capture local image structure beyond simple pixel intensities.
  • Mid-level features: This level utilizes image segmentation. By grouping pixels into regions (superpixels), the method reduces computational complexity and mitigates the influence of large, uniform color areas that can skew estimations. A hierarchical segmentation method is employed.
  • High-level features: This is a key novelty. The authors incorporate scene context (e.g., indoor/outdoor) and stage models (representing scene geometry or object types). These high-level cues provide strong priors about the likely illuminants, allowing the algorithm to narrow down possibilities and improve accuracy. For instance, outdoor scenes are typically illuminated by daylight variants.

The Bayesian framework fuses these cues by learning empirical distributions for reflectances and illuminants from training data, allowing the model to adapt to different image characteristics.

Datasets

The experiments were conducted on two datasets:

  • Gehler color constancy dataset: Reprocessed by Shi and Funt [24]. This dataset contains 568 images.
  • Dataset reprocessed by Lynch et al. [25]: This dataset is also used for evaluation.

The paper does not introduce a new dataset but rather re-evaluates existing ones using their proposed method.

Key Insights and Ideas

  • The power of multi-level fusion: Combining low-level, mid-level, and high-level cues is significantly more effective than using any single cue or even just low-level combinations.
  • High-level context is crucial: Scene understanding (indoor/outdoor) and geometric cues (stage models) provide strong priors that dramatically improve color constancy, especially by constraining the possible illuminant colors.
  • Bayesian framework as a unifying principle: The Bayesian approach offers a principled and flexible way to integrate diverse information sources, allowing for easy addition of new cues.
  • Mid-level cues (regions) are highly informative: Segmentation into regions not only reduces computation but also provides a more robust representation than raw pixels, especially when dealing with large uniform areas.
  • Higher-order statistics (n-jet) offer an advantage over simple pixel values: Capturing local image structure through derivatives improves performance.

Key Results

  • The proposed method significantly outperforms the original Bayesian baseline. For the Shi and Funt dataset, the mean error is reduced by 33.3% and the median error by 37.1%.
  • The method achieves state-of-the-art performance, outperforming most other algorithms in mean angular error and obtaining the highest accuracy in median angular error on the tested datasets.
  • Combining all cues (pixel, 1-jet, 2-jet, region) consistently yields the best results across different scenarios.
  • Using indoor/outdoor classification improves performance for all tested methods, highlighting the importance of scene context.
  • The paper demonstrates that mid-level cues (regions) are particularly important, with combinations involving regions showing substantial improvements.

Limitations and Future Work

  • Manual annotation for stages: While the paper explores stage models, it relies on manual annotation for these geometric cues. The performance with automatically detected stages is discussed but not as extensively evaluated as manual ones.
  • Computational cost: While segmentation reduces complexity, fusing multiple cues and running a Bayesian inference can still be computationally intensive.
  • Future Work: The authors suggest incorporating other high-level information, such as object detection, to further enhance color constancy results. They also plan to explore more sophisticated ways to integrate different cues and potentially develop more efficient inference methods.

Summary updated

BibTeX
@article{liu2015color,
  author = {Yang Liu and Theo Gevers and Xueqing Li},
  title = {{Color constancy by combining low-mid-high level image cues}},
  journal = {Comput. Vis. Image Underst.},
  year = {2015},
  doi = {10.1016/j.cviu.2015.04.011},
}
Figure