Abstract
Accurately reporting what objects are depicted in an image is largely a solved problem in automatic caption generation. The next big challenge on the way to truly humanlike captioning is being able to incorporate the context of the image and related real world knowledge. We tackle this challenge by creating an end-to-end caption generation system that makes extensive use of image-specific encyclopedic data. Our approach includes a novel way of using image location to identify relevant open-domain facts in an external knowledge base, with their subsequent integration into the captioning pipeline at both the encoding and decoding stages. Our system is trained and tested on a new dataset with naturally produced knowledge-rich captions, and achieves significant improvements over multiple baselines. We empirically demonstrate that our approach is effective for generating contextualized captions with encyclopedic knowledge that is both factually accurate and relevant to the image.
Introduction: Bridging the Context Gap
Standard image captioning models have become proficient at identifying visual components—such as a "clock tower" or a "bridge"—but they often lack the human-like ability to provide historical or cultural context. To bridge this "context gap," models require access to external, open-domain knowledge. While previous attempts relied on visual object recognition to query databases, this research proposes using geographic metadata (GPS) as a precise grounding anchor to retrieve specific facts about landmarks and entities from databases like DBpedia and OpenStreetMap.
System Architecture: Anchoring Pixels to Facts
The proposed system utilizes an end-to-end encoder-decoder architecture based on Transformers. The core innovation lies in the integration of external knowledge at multiple stages of the pipeline.
The process begins by using an image's GPS coordinates to query OpenStreetMap for nearby entities. This "Geographic Context" is used to generate embeddings (GEOEMB) based on distance, direction, and size. Subsequently, the system queries DBpedia for related facts in the form of triples (e.g., <subject, predicate, object>). These facts are ranked and converted into "Fact Embeddings" (FACTEMB). During decoding, the model uses "Predicate Indicators" and "Geo Entity Indicators" to ensure that the generated text correctly incorporates these external data points into a grammatically sound structure.
The Geograph Dataset
To train and evaluate the model, the researchers introduced the Geograph Dataset, consisting of 7,128 images from the Geograph Britain and Ireland project. These images are uniquely suited for the task as they contain naturally produced, knowledge-rich captions and precise location metadata. To ensure robust evaluation, the data was split by latitude, preventing the model from simply memorizing landmarks from specific training regions.
Experimental Results
The "Knowledge-from-metadata" approach significantly outperformed standard visual-only baselines and models relying solely on visual-based knowledge retrieval.
As shown in the results, the system achieved a CIDEr score of 32.41, a massive improvement over the "No-knowledge" baseline (1.75) and the standard ClipCap model (0.14). Beyond standard metrics, the model demonstrated high factual reliability, achieving 86.08% factual accuracy when verified against DBpedia. This allows the system to generate detailed captions such as: "kelso bridge. the bridge over the river tweed was built in 1800, and was designed by john rennie the elder."
Conclusion
By using GPS as a primary key for knowledge retrieval, this system overcomes the limitations of visual-only recognition. While currently dependent on structured metadata, the success of this approach opens doors for context-aware captioning in other specialized fields, such as medical imaging or historical archiving, where metadata can be used to query expert databases.
Summary updated