What is the Color of Chocolate? - Extracting Color Values of Semantic Expressions

Albrecht J. Lindner, Nicolas Bonnier, Sabine Süsstrunk

  • Extracting Color Values of Semantic Expressions

Introduction

Researchers have developed a fully automated framework that maps semantic expressions—ranging from standard color names like "crimson" to objects like "chocolate" or brands like "Ferrari"—to their corresponding digital color values. By statistically mining millions of tagged images on Flickr, the system bridges the "semantic gap" between human language and RGB triplets without requiring a single human subject or manual experiment.

The Semantic Gap

In computer science, there is a persistent "semantic gap" between how humans describe the world and how computers represent it. To a human, "chocolate" evokes a specific shade of brown; to a computer, that color is an abstract triplet like $(98, 55, 32)$.

Traditionally, bridging this gap required expensive and time-consuming psychophysical experiments: bringing people into a lab, showing them color swatches, and recording their responses. This process is limited to a small set of words. The authors sought a way to leverage the "wisdom of crowds" to automatically learn these associations for any arbitrary string of text.

Methodology: Mining the Wisdom of Crowds

The authors treat the problem as a translation task between "semantic language" (tags) and "numeric language" (pixels).

  1. Data Acquisition: The system utilizes the Flickr API to download images and their associated metadata (tags).
  2. Color Quantization: Images are converted to the CIELAB color space, which is designed to be perceptually uniform. The space is divided into a $15 \times 15 \times 15$ grid of histogram bins.
  3. Statistical Analysis: To determine if a keyword (e.g., "smurf") is associated with a color, the framework employs the Mann-Whitney-Wilcoxon rank-sum test. It compares images containing the keyword against those without it to see if specific color bins are significantly more populated.
  4. The z-score: The result is a $z$-score for every color bin. The bin with the highest $z$-score identifies the representative color, while the magnitude of the score indicates the association strength—how strongly that word is tied to a specific color.

Validation and Benchmarking

To ensure the automated results align with human perception, the system was validated against two primary benchmarks:

  • Color Naming Benchmarks: The model was compared against Moroney’s online color naming experiment. For the 50 most common color names, the system's estimates were remarkably close to human-derived data, with most differences falling within a $\Delta E$ (color difference) of 8.2.
  • Memory Color Benchmarks: The authors compared their results against Yendrikhovskij’s ellipses, a standard dataset for "memory colors" (colors of familiar objects like skin, sky, and grass).
Validation of extracted memory colors against established psychophysical ellipses.

Insights and Discussion

Beyond Color Names

The algorithm successfully identifies colors for non-color words. It correctly determined that "Ferrari" is a specific high-significance red, while "Lakers" yielded the team's iconic violet and yellow. This demonstrates the system's ability to extract cultural and brand-specific color associations.

Extracted color values for a wide range of semantic expressions, from standard names to brands and objects.

Contextual Nuance

The system can handle complex queries. By combining keywords (e.g., "vegetation + dry" vs. "vegetation + wet"), the framework identifies distinct shifts in hue and saturation that match human expectations of how environment affects color.

The "Puce" Problem and Semantic Ambiguity

The study revealed how polysemy (multiple meanings) affects data. The system initially failed on "puce" because, on Flickr, the term often refers to a specific band or microchips, leading to a dark/black result rather than the traditional purplish-brown. This highlights a limitation: the results are a reflection of what people choose to photograph and tag, which may not always represent the "ground truth" of a color name.

Conclusion

This framework provides a scalable, automated alternative to traditional psychophysical experiments. By quantifying the "colorfulness" of concepts through $z$-scores, it offers a new way to bridge the gap between human language and digital representation. Future work may integrate word-sense disambiguation to handle semantic ambiguity and improve automated image enhancement tools.

Summary updated

BibTeX
@inproceedings{lindner2012what,
  author = {Albrecht J. Lindner and Nicolas Bonnier and Sabine Süsstrunk},
  title = {{What is the Color of Chocolate? - Extracting Color Values of Semantic Expressions}},
  booktitle = {Conference},
  year = {2012},
  doi = {10.2352/cgiv.2012.6.1.art00062},
}
Figure