Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models

Pranav Agarwal, Alejandro Betancourt, Vana Panagiotou, Natalia Díaz RodríguezarXiv 2020 · arXiv

Abstract

Current image captioning models often suffer from a "narrow vision" bias, limited by the small number of object classes in standard training datasets. This paper introduces Egoshots, a dataset of 978 unconstrained first-person life-logging images, and Semantic Fidelity (SF), a novel evaluation metric that does not require ground-truth human annotations. By utilizing a high-capacity object detector as a semantic oracle, we demonstrate that state-of-the-art models fail to capture the rich diversity of real-world scenes. Our results show that the SF metric correlates strongly with human judgment ($\rho = 0.93$), providing a scalable way to evaluate models on "in-the-wild" data.

Introduction

Most modern image captioning models are trained on datasets like MSCOCO or Pascal-VOC, which contain only 80 and 20 object classes, respectively. While these models generate grammatically correct sentences, they often ignore the variety of items found in everyday life, focusing only on the handful of objects they were trained to recognize. Furthermore, evaluating performance on new, unlabeled data is challenging because standard metrics (e.g., BLEU, CIDEr) require human-written reference captions. This paper addresses these gaps by providing a new benchmark dataset and a reference-free metric.

The Egoshots Dataset

The authors collected Egoshots, a dataset comprising 978 ego-vision images captured using an Autographer wearable camera. Unlike curated datasets, Egoshots contains diverse, unconstrained real-life situations such as office work, biking, and socializing. These images present significant challenges for computer vision, including varied lighting, motion blur, and unusual camera angles.

Semantic Fidelity (SF) Metric

To evaluate captions without reference labels, the authors propose the Semantic Fidelity (SF) metric. This metric uses a powerful Object Detector (OD)—specifically YOLO9000, capable of recognizing 9,000 classes—as a proxy for ground truth.

Overview of the Semantic Fidelity (SF) metric pipeline.

The SF score for a caption is calculated by combining semantic similarity and a diversity ratio:

$$SF_i = s_i \cdot \frac{\#N}{\#O}$$

Where:

  • $s_i$ (Semantic Similarity): Measures the proximity of nouns in the generated caption to the objects detected by the OD using word embeddings (e.g., Word2Vec).
  • $\#N / \#O$ (Diversity Ratio): The ratio of unique nouns in the caption to unique objects detected in the image, penalizing overly simplistic descriptions.

Key Insights and Results

The "Lazy" Captioner

Analysis using the SF metric reveals a significant gap in descriptive capability. While YOLO9000 identifies an average of 5–7 objects per image in Egoshots, captioning models typically mention only 2–3. This suggests that models are "lazy," defaulting to a few common classes rather than describing the full scene.

Model Benchmarking and Robustness

The authors benchmarked several state-of-the-art models, including Show Attend and Tell (SAT) and nocaps (NOC). While models designed for novel objects (NOC and DNOC) outperformed standard baselines, they still lagged significantly behind the object diversity captured by the detector. Additionally, when tested on surreal or "impossible" images, models often "hallucinated" common objects (e.g., describing a surreal painting of a man with a bicycle as "a man sitting on a bed with a cat") because the scenes deviated from training distributions.

Human Correlation

The SF metric demonstrated a high Pearson correlation ($\rho = 0.93$) with human-annotated semantic fidelity scores on MSCOCO subsets. This validates SF as a reliable, automated proxy for human judgment in scenarios where reference captions are unavailable.

Limitations and Future Work

While promising, the SF metric currently focuses primarily on nouns and does not account for verbs, attributes (e.g., color, size), or grammatical correctness. It also lacks the ability to distinguish between multiple instances of the same object. Future work will involve extending the metric to include verb-based semantic similarity and integrating more sophisticated counting mechanisms. Furthermore, the metric's accuracy remains dependent on the performance of the underlying object detector used as the oracle.

Summary updated

Abstract

Image captioning models have been able to generate grammatically correct and human understandable sentences. However most of the captions convey limited information as the model used is trained on datasets that do not caption all possible objects existing in everyday life. Due to this lack of prior information most of the captions are biased to only a few objects present in the scene, hence limiting their usage in daily life. In this paper, we attempt to show the biased nature of the currently existing image captioning models and present a new image captioning dataset, Egoshots, consisting of 978 real life images with no captions. We further exploit the state of the art pre-trained image captioning and object recognition networks to annotate our images and show the limitations of existing works. Furthermore, in order to evaluate the quality of the generated captions, we propose a new image captioning metric, object based Semantic Fidelity (SF). Existing image captioning metrics can evaluate a caption only in the presence of their corresponding annotations; however, SF allows evaluating captions generated for images without annotations, making it highly useful for real life generated captions.
BibTeX
@misc{agarwal2020egoshots,
  author = {Pranav Agarwal and Alejandro Betancourt and Vana Panagiotou and Natalia Díaz Rodríguez},
  title = {{Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models}},
  year = {2020},
  eprint = {2003.11743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.11743},
}
Figure