Abstract
In this paper we tackle the problem of automatic caption generation for news images. Our approach leverages the vast resource of pictures available on the web and the fact that many of them are captioned. Inspired by recent work in summarization, we propose extractive and abstractive caption generation models. They both operate over the output of a probabilistic image annotation model that pre-processes the pictures and suggests keywords to describe their content. Experimental results show that an abstractive model defined over phrases is superior to extractive methods.
Introduction
Computer vision research has traditionally focused on image annotation—the task of assigning descriptive keywords to images. While identifying elements like "sky" or "car" is useful, these tags lack the relational context necessary for human-like understanding. For instance, keywords cannot distinguish between a "blue car" and a "blue sky." In the news domain, the challenge is amplified: captions must synthesize visual information with specific contextual details (names, dates, events) found in the accompanying article. This paper addresses the gap between simple tagging and context-aware natural language generation by proposing a model that generates full-sentence captions for news images.
Methodology
The proposed framework consists of a two-stage pipeline:
1. Image Annotation
The system first analyzes the image using SIFT (Scale Invariant Feature Transform) descriptors. These visual features are combined with text using Latent Dirichlet Allocation (LDA), treating images and their associated articles as a joint "bag of words." This allows the model to predict the most relevant keywords for a given picture based on its visual content and the document context.
2. Caption Generation
Using the predicted keywords as a filter, the authors evaluated two primary strategies:
- Extractive Models: These models search the news article for the single sentence that best matches the predicted image keywords, utilizing similarity metrics such as Cosine Similarity or KL Divergence.
- Abstractive Models: These models synthesize new sentences. The authors developed a word-based model and a more sophisticated phrase-based model. The phrase-based approach extracts linguistically motivated chunks (heads and dependents) from the article and uses a "phrase attachment" model to determine their optimal ordering and combination, ensuring better syntactic coherence.
Experimental Setup
The models were trained and tested on a dataset of 3,361 articles and captioned images scraped from the BBC News website. A significant challenge noted in this dataset is that over 50% of the words in gold-standard captions do not appear in the main article text, indicating that journalists often bring in external knowledge not present in the primary story.
Results and Discussion
Evaluation was conducted using both the Translation Edit Rate (TER)—where lower scores indicate better performance—and human judgments on a 1–7 scale for grammaticality and relevance.
As shown in the quantitative results, the Abstractive Phrase-based model significantly outperformed all other methods, achieving a TER of 1.06. This is a substantial improvement over the Lead Sentence baseline (2.12) and the best extractive model (KL Divergence at 1.77).
Human evaluations further validated these findings. The phrase-based model achieved scores of 4.80 for grammaticality and 4.96 for relevance. While it did not reach the performance of human-written "Gold Standard" captions, it was perceived as significantly more relevant than extractive methods. The results also highlighted the importance of visual features; models that did not utilize image keywords performed poorly, confirming that the computer vision component is essential for selecting relevant text.
Conclusion
This research demonstrates that abstractive, phrase-based models are highly effective for generating news captions. By using visual keywords to filter document content and linguistically motivated phrases to construct sentences, the system produces captions that are both relevant and grammatical. Future work will focus on deeper syntactic integration and end-to-end models that unify image annotation and caption generation.
Summary updated