A Poodle or a Dog? Evaluating Automatic Image Annotation Using Human Descriptions at Different Levels of Granularity

Josiah Wang, Fei Yan, Ahmet Aker, Robert J. GaizauskasVL @ COLING 2014 · OpenReview

TL;DR

This paper tackles the ambiguity in automatic image annotation by introducing a "granularity-aware" approach. By grouping semantically related concepts (e.g., "poodle" and "dog") at different levels of specificity, they improve annotation flexibility and human-centric evaluation, showing that this approach can lead to more informative image descriptions.

What Problem Does This Paper Address?

Automatic image annotation systems can recognize thousands of object categories, but they often struggle with the inherent ambiguity of human language. For instance, an image of a dog could be labeled as "dog," "poodle," "puppy," "pet," or even "animal." Current systems typically output a ranked list of labels, but the definition of a "correct" label is often unclear, especially when multiple levels of granularity are equally valid. This paper addresses the challenge of how to evaluate and improve image annotation systems to better align with how humans naturally describe image content, acknowledging that what one person considers a "basic-level" category might differ for another.

Methods

The core of this paper is the "granularity-aware" approach, which involves grouping semantically related concepts. This is achieved by:

  • Semantic Grouping: Exploiting the hypernym/hyponym hierarchy of WordNet to group concepts. They adapt a "word naturalness" measure, inspired by previous work, to determine how likely a person is to use a particular term to describe an image. This measure balances the "naturalness" of a term with its specificity, allowing for flexible grouping.
  • Re-ranking Visual Classifier Output: The grouped concepts are then used to re-rank the output of visual classifiers. Instead of considering individual synset scores, the paper proposes taking the maximum score within a semantic group as the group's score. This allows for a more flexible and potentially more human-aligned annotation.
  • Evaluation with Human Descriptions: To evaluate their approach, the authors use human-written captions from a dataset. They automatically extract nouns from these captions to create a "gold standard" annotation, which is then used to compare the performance of their re-ranked annotations against the initial, un-grouped ones.

Datasets

  • ImageNet: Used for training visual classifiers.
  • Flickr8k dataset: Used for evaluation. This dataset contains 8,091 images, each annotated with five textual descriptions. The authors specifically chose this dataset to demonstrate granularity in large-scale object hierarchies and limited their categories to synsets found in the textual descriptions to manage computational complexity.

Key Insights and Ideas

  • Human-centricity is Key: The paper strongly advocates for evaluating image annotation systems from a human perspective, recognizing that human language is inherently flexible and context-dependent.
  • Granularity Matters: The "poodle vs. dog" example highlights that a single "correct" label is often insufficient. Grouping concepts across different levels of specificity is crucial for better representation.
  • "Naturalness" as a Guiding Principle: Using a measure of how "natural" a term is for humans to use provides a principled way to define semantic groups and balance specificity with generality.
  • Flexibility vs. Informativeness Trade-off: While increased flexibility through broader groupings can improve scores, there's a risk of becoming too general and thus less informative (e.g., grouping everything under "being").

Key Results

  • The proposed "granularity-aware" groupings consistently improve the Normalized Discounted Cumulative Gain (NDCG) scores compared to random baseline groupings.
  • As the parameter controlling group flexibility (λ) increases, NDCG scores generally increase, indicating that broader groupings can be more effective.
  • However, the paper notes that very high flexibility can lead to less informative annotations, suggesting a sweet spot for granularity. Figure 1 provides a visual example of this re-ranking process, showing how different values of λ affect the ranked labels.

Limitations and Future Work

  • Subjectivity of Informativeness: The paper acknowledges that the "informativeness" of groupings is subjective and context-dependent, posing an interesting open question.
  • Future Work: The authors suggest several avenues for future research:
  • Exploring different methods for grouping concepts.
  • Incorporating visual classifier outputs to improve the grouping process itself.
  • Leveraging more textual sources to enhance image annotation.
  • Extending the approach to generate full sentence annotations rather than just lists of labels.

Summary updated

BibTeX
@inproceedings{wang2014poodle,
  author = {Josiah Wang and Fei Yan and Ahmet Aker and Robert J. Gaizauskas},
  title = {{A Poodle or a Dog? Evaluating Automatic Image Annotation Using Human Descriptions at Different Levels of Granularity}},
  booktitle = {VL @ COLING},
  year = {2014},
  url = {https://openreview.net/forum?id=BepifgxJRm},
}

Loading related content...

Figure