Enhancing bag of visual words with color information for iconic image classification

Stephan Kopf, Mariia Zrianina, Benjamin Guthier, Lydia Weiland, Philipp Schaber, Simone Paolo Ponzetto, Wolfgang EffelsbergMADOC 2017 · OpenAlex

Introduction

In the field of computer vision, "iconic images" serve as powerful symbols representing abstract, culturally understood concepts. For instance, a lone polar bear on an ice floe is not merely a biological specimen; it is a widely recognized icon for climate change. Traditional image classification frameworks, such as the Bag of Visual Words (BoVW), often struggle with these images because they rely heavily on texture-based descriptors like SIFT or SURF, which are typically grayscale and ignore color information.

What Problem Does This Paper Address?

The core challenge lies in the fact that for many iconic images, color is the defining characteristic rather than texture. A field of green grass and a field of golden wheat might appear nearly identical to a grayscale algorithm, yet they represent distinct categories with different symbolic meanings.

Iconic images representing contrasting environmental concepts: industrial pollution versus renewable energy.

Standard BoVW methods detect keypoints based on corners or edges and describe their local texture. By ignoring color, these methods lose a critical dimension of semantic information necessary for identifying symbolic content.

Methods

To bridge this gap, the researchers introduced a Color Pyramid feature integrated into the traditional BoVW pipeline. This approach enhances the system's ability to "see" in color by filtering local descriptors based on their hue.

The proposed method follows these steps:

  1. Standard Pipeline: The system identifies keypoints using SIFT or SURF and clusters them into a visual "vocabulary."
  2. Color Masking: Images are converted to the HSV (Hue, Saturation, Value) color space. A series of masks are created based on specific Hue ranges (e.g., blue, orange, green).
  3. Keypoint Filtering: Only keypoints falling within a specific color mask are included in that color's specific histogram.
  4. Feature Concatenation: Histograms from each color mask are stitched together into a single, high-dimensional feature vector. This allows the classifier—typically a Support Vector Machine (SVM)—to associate specific textures with specific colors.
The BoVW_qt_m11 software interface showing the training and cross-validation settings, including the 'Color Pyramids' feature.

Experimental Setup

The researchers evaluated their method using a custom dataset of 1,500 images categorized into 15 "iconic" topics, including "North Nature" (Aurora, Clouds), "Agriculture" (Tractor, Wheat), and "Africa Nature" (Giraffe, Elephant). These images were sourced from Flickr and manually curated to ensure they represented the intended symbolic concepts.

Key Results and Insights

The inclusion of color pyramids resulted in a significant performance boost, with an average increase in the F1 measure of 0.118.

  • Texture vs. Color: The most notable improvements occurred in categories where texture alone was ambiguous. For example, the system previously confused "aurora" with "clouds" due to their similar wispy textures. Adding color information (green/purple vs. white/blue) effectively eliminated this confusion.
  • Category Performance: The "Giraffe" category saw the highest improvement (+0.144 F1) due to its distinct coloration. Conversely, the "Elephant" category showed minimal gain (+0.0039 F1), as elephants are largely monochromatic and texture remains their primary identifier.
  • Computational Trade-offs: While effective, the color pyramid feature increases computational complexity. Using SIFT with color pyramids increased training time by approximately six times, though this was partially mitigated when using the faster SURF descriptors.

Conclusion

The research demonstrates that color is an essential, "orthogonal" feature that significantly enhances the classification of iconic images. By mimicking the human ability to categorize objects by both shape and color, the enhanced BoVW framework provides a more robust tool for understanding symbolic multimedia content. Future work will explore multimodal approaches, combining these visual enhancements with textual analysis of image metadata.

Summary updated

BibTeX
@inproceedings{kopf2017enhancing,
  author = {Stephan Kopf and Mariia Zrianina and Benjamin Guthier and Lydia Weiland and Philipp Schaber and Simone Paolo Ponzetto and Wolfgang Effelsberg},
  title = {{Enhancing bag of visual words with color information for iconic image classification}},
  booktitle = {MADOC},
  year = {2017},
}
Figure