Automatic Image Annotation Using Auxiliary Text Information

Yansong Feng, Mirella LapataACL 2008 · OpenReview

Abstract

The availability of databases of images labeled with keywords is necessary for developing and evaluating image annotation models. Dataset collection is however a costly and time consuming task. In this paper we exploit the vast resource of images available on the web. We create a database of pictures that are naturally embedded into news articles and propose to use their captions as a proxy for annotation keywords. Experimental results show that an image annotation model can be developed on this dataset alone without the overhead of manual annotation. We also demonstrate that the news article associated with the picture can be used to boost image annotation performance.

Introduction: The Data Bottleneck in Image Annotation

Automatic image annotation—assigning relevant keywords to visual content—is essential for managing large-scale digital collections. Traditionally, research has relied on curated datasets like the Corel database. While "clean," Corel is limited by its small size and idealized nature, where images typically feature single, centered objects. Real-world web images are far more complex and lack manual labels. This paper addresses the "data bottleneck" by leveraging the vast, naturally annotated environment of online news, using captions and full-text articles as proxies for manual annotation.

Methodology: Leveraging Auxiliary Text

The authors extend the Continuous Relevance Model (CRM), a probabilistic framework for joint image-word distribution, with three key innovations:

  1. Multiple Bernoulli Distribution: By shifting from a Multinomial to a Multiple Bernoulli approach, the model focuses on the presence or absence of words. This is better suited for the concise and varied nature of news captions.
  2. Auxiliary Text Integration: The model incorporates the entire vocabulary of the news article, not just the caption. A weighted combination allows the system to predict words found in the broader context of the story.
  3. LDA Re-ranking: Latent Dirichlet Allocation (LDA) is used to filter candidate keywords. By identifying the "gist" of the article, LDA re-ranks words to ensure they align with the topical context, effectively de-noising the output.

The BBC News Database

To test their approach, the authors developed the BBC News Database, consisting of 3,361 image-caption-document tuples. Unlike the Corel set, which has a vocabulary of roughly 260 words, the BBC set features over 2,100 words, reflecting the complexity of real-world reporting.

Example of a BBC News article used in the dataset, featuring an image, caption, and auxiliary text.

The dataset highlights the difference between "object" annotation (identifying a tree) and "semantic" annotation (understanding an event like a harvest or a protest).

Key Insights and Results

The research validates that news captions are high-quality proxies; approximately 90% of caption content words describe the image. Furthermore, the full article provides the "denotative" context (who, what, where) that anchors brief or connotative captions.

Sample images from the BBC News database representing diverse real-world subjects.

The proposed ExtModel significantly outperformed standard CRM baselines:

  • F1 Score: The ExtModel achieved 19.82% on the BBC dataset, nearly doubling the original CRM's 11.81%.
  • Article Boost: Including the full news article text improved the F1 score by over 8% compared to using captions alone.

Conclusion and Future Directions

This work demonstrates that "noisy" web data can be a powerful resource for training annotation models without manual overhead. Future improvements could involve deeper text processing, such as Named Entity Recognition (NER), and higher-resolution visual analysis to better decode the complex scenes typical of global news.

Summary updated

Abstract

The availability of databases of images labeled with keywords is necessary for developing and evaluating image annotation models. Dataset collection is however a costly and time consuming task. In this paper we exploit the vast resource of images available on the web. We create a database of pictures that are naturally embedded into news articles and propose to use their captions as a proxy for annotation keywords. Experimental results show that an image annotation model can be developed on this dataset alone without the overhead of manual annotation. We also demonstrate that the news article associated with the picture can be used to boost image annotation performance.
BibTeX
@inproceedings{feng2008automatic,
  author = {Yansong Feng and Mirella Lapata},
  title = {{Automatic Image Annotation Using Auxiliary Text Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2008},
  url = {https://openreview.net/forum?id=S1MrMhx_-H},
}
Figure