Abstract
Current image captioning models often suffer from a "narrow vision" bias, limited by the small number of object classes in standard training datasets. This paper introduces Egoshots, a dataset of 978 unconstrained first-person life-logging images, and Semantic Fidelity (SF), a novel evaluation metric that does not require ground-truth human annotations. By utilizing a high-capacity object detector as a semantic oracle, we demonstrate that state-of-the-art models fail to capture the rich diversity of real-world scenes. Our results show that the SF metric correlates strongly with human judgment ($\rho = 0.93$), providing a scalable way to evaluate models on "in-the-wild" data.
Introduction
Most modern image captioning models are trained on datasets like MSCOCO or Pascal-VOC, which contain only 80 and 20 object classes, respectively. While these models generate grammatically correct sentences, they often ignore the variety of items found in everyday life, focusing only on the handful of objects they were trained to recognize. Furthermore, evaluating performance on new, unlabeled data is challenging because standard metrics (e.g., BLEU, CIDEr) require human-written reference captions. This paper addresses these gaps by providing a new benchmark dataset and a reference-free metric.
The Egoshots Dataset
The authors collected Egoshots, a dataset comprising 978 ego-vision images captured using an Autographer wearable camera. Unlike curated datasets, Egoshots contains diverse, unconstrained real-life situations such as office work, biking, and socializing. These images present significant challenges for computer vision, including varied lighting, motion blur, and unusual camera angles.
Semantic Fidelity (SF) Metric
To evaluate captions without reference labels, the authors propose the Semantic Fidelity (SF) metric. This metric uses a powerful Object Detector (OD)—specifically YOLO9000, capable of recognizing 9,000 classes—as a proxy for ground truth.
The SF score for a caption is calculated by combining semantic similarity and a diversity ratio:
$$SF_i = s_i \cdot \frac{\#N}{\#O}$$
Where:
- $s_i$ (Semantic Similarity): Measures the proximity of nouns in the generated caption to the objects detected by the OD using word embeddings (e.g., Word2Vec).
- $\#N / \#O$ (Diversity Ratio): The ratio of unique nouns in the caption to unique objects detected in the image, penalizing overly simplistic descriptions.
Key Insights and Results
The "Lazy" Captioner
Analysis using the SF metric reveals a significant gap in descriptive capability. While YOLO9000 identifies an average of 5–7 objects per image in Egoshots, captioning models typically mention only 2–3. This suggests that models are "lazy," defaulting to a few common classes rather than describing the full scene.
Model Benchmarking and Robustness
The authors benchmarked several state-of-the-art models, including Show Attend and Tell (SAT) and nocaps (NOC). While models designed for novel objects (NOC and DNOC) outperformed standard baselines, they still lagged significantly behind the object diversity captured by the detector. Additionally, when tested on surreal or "impossible" images, models often "hallucinated" common objects (e.g., describing a surreal painting of a man with a bicycle as "a man sitting on a bed with a cat") because the scenes deviated from training distributions.
Human Correlation
The SF metric demonstrated a high Pearson correlation ($\rho = 0.93$) with human-annotated semantic fidelity scores on MSCOCO subsets. This validates SF as a reliable, automated proxy for human judgment in scenarios where reference captions are unavailable.
Limitations and Future Work
While promising, the SF metric currently focuses primarily on nouns and does not account for verbs, attributes (e.g., color, size), or grammatical correctness. It also lacks the ability to distinguish between multiple instances of the same object. Future work will involve extending the metric to include verb-based semantic similarity and integrating more sophisticated counting mechanisms. Furthermore, the metric's accuracy remains dependent on the performance of the underlying object detector used as the oracle.
Summary updated