BossaNova at ImageCLEF 2012 Flickr Photo Annotation Task

Sandra Eliza Fontes de Avila, Nicolas Thome, Matthieu Cord, Eduardo Valle, Arnaldo de Albuquerque Araújo

TL;DR

This paper introduces "BossaNova," a novel mid-level image representation that enhances the traditional Bag-of-Words approach by incorporating distance information between image features and codebook entries. This method achieved a strong second-place ranking in the ImageCLEF 2012 Flickr Photo Annotation Task, demonstrating its effectiveness and efficiency.

What Problem Does This Paper Address?

Image annotation, the task of automatically assigning descriptive tags to images, is a fundamental challenge in computer vision. Traditional methods like Bag-of-Words (BoW) represent images by summarizing local features (like SIFT descriptors) into a histogram of "codewords" from a learned dictionary. While effective, BoW loses information about the spatial relationships and precise distances of these features to the codewords. This paper addresses the limitation of BoW by proposing a more informative representation that captures these finer details, aiming to improve image classification performance.

Methods

The core contribution is the BossaNova representation. It builds upon the BoW framework but modifies the pooling step. Instead of just counting the presence of codewords, BossaNova creates a histogram of distances between local image descriptors and each codeword in the codebook. This provides a richer representation by capturing not just which features are present, but also how far they are from their closest learned representations.

The BossaNova scheme involves three main steps:

  1. Local Feature Extraction: SIFT (Scale-Invariant Feature Transform) descriptors are extracted from images.
  2. BossaNova Representation: This is the novel part. For each codeword in the learned codebook, a histogram of distances to the image's local descriptors is computed. This histogram is then concatenated with a simple count of how many descriptors are associated with that codeword (a BoW-like term).
  3. Classification: Support Vector Machines (SVMs) are used to classify images based on their BossaNova representations.

The authors highlight that BossaNova is conceptually simple, non-parametric, and easily adaptable. It also leads to more compact representations compared to other BoW enrichment methods and complements well with advanced techniques like Fisher Vectors.

Datasets

The primary dataset used is the ImageCLEF 2012 Flickr Photo Annotation Task dataset. This dataset consists of:

  • 25,000 Flickr images.
  • A training set of 15,000 images.
  • A test set of 10,000 images.
  • 94 diverse concepts for annotation, covering categories like people, nature, weather, and sentiments.

The authors also mention using a subset of one million randomly sampled descriptors for learning codebooks (via k-means) and training Gaussian Mixture Models (GMMs) for Fisher Vectors.

Key Insights and Ideas

  • Distance Matters: The key insight is that the distance of local features to learned visual words (codewords) is a crucial piece of information that is lost in standard Bag-of-Words. Capturing this distance distribution enriches the image representation significantly.
  • Compactness and Efficiency: BossaNova offers a more compact representation than some other mid-level representations, which is beneficial for large-scale image classification.
  • Complementary to Fisher Vectors: The paper demonstrates that BossaNova can be effectively combined with Fisher Vectors, leading to even better performance than either method alone. This suggests a synergistic relationship between these different levels of image representation.
  • Simplicity and Adaptability: The method's straightforward nature makes it easy to implement and adapt to different datasets and feature types.

Key Results

The BossaNova method was evaluated in the ImageCLEF 2012 Flickr Photo Annotation Task, where the authors submitted four purely visual runs.

  • Their best performing run (Run 1), which combined BossaNova with Fisher Vector representations using SIFT descriptors, achieved a Mean interpolated Average Precision (MiAP) of 34.37%.
  • This result secured them the second rank among 28 purely visual submissions and 18 participating teams.
  • The BossaNova method alone (Run 3) also performed very well, achieving the third rank with a MiAP of 33.64%.

Limitations and Future Work

  • Opponent SIFT Performance: The paper notes that using Opponent SIFT descriptors in combination with BossaNova (Run 4) did not yield improvements. They attribute this to the severe dimensionality reduction applied to these descriptors, suggesting that preserving more information from these features might be beneficial.
  • Combination Strategy: While combining BossaNova and Fisher Vectors via a weighted sum of kernel functions was effective, the authors mention that concatenating the feature vectors directly (as done in prior work) yielded slightly better results. This indicates potential for further optimization in how different representations are fused.
  • Feature Combinations: The authors explicitly state that they are currently investigating feature combinations in a kernel learning framework to leverage all available features more effectively. This suggests that exploring more diverse low-level descriptors and sophisticated fusion techniques is a promising future direction.

Summary updated

BibTeX
@inproceedings{avila2012bossanova,
  author = {Sandra Eliza Fontes de Avila and Nicolas Thome and Matthieu Cord and Eduardo Valle and Arnaldo de Albuquerque Araújo},
  title = {{BossaNova at ImageCLEF 2012 Flickr Photo Annotation Task}},
  booktitle = {CLEF (Online Working Notes/Labs/Workshop)},
  year = {2012},
}
Figure